Chapter 25 · Operate the capstone as a reproducible, diagnosable robot service
Today in the field story
One problem, then the next
Before final trials, the train freezes unseen object positions, lighting states, and distractors with stable reset, seed, outcome, timeout, intervention, and exclusion rules. Deterministic regressions and randomized robustness exploration remain separate rows. Conditions revealed during Week 24's held-out evaluation are historical, not unseen again. Any new failure becomes evidence for a later release rather than permission to tune Model v1 midway. This keeps the forthcoming fifty trials a test of the frozen service, not an interactive debugging session.
- Why now
A frozen matrix protects the meaning of robustness claims.
- Ignore today
Do not peek, rebalance, or tune against final scenarios.
- Unlocks next
An adjudicable 50-trial manifest with no hidden condition changes.
Understand
Build the physical picture first
Held-out testing is a surprise obstacle course whose layout is sealed before the robot enters, while each changed physical factor stays labeled for later diagnosis.
An unseen condition is outside the examples used to fit the model or tune the system, but “unseen” must name a boundary. A new object position may still lie inside the trained workspace; a new lamp direction may change shadows; a distractor may resemble the target. Reserve evaluation scenes before tuning, store their identities separately, and disclose prior exposure from pretraining or manual inspection when known. Novelty is graded evidence, not a magical claim that the robot understands every new room.
Turn variation into factors and levels. For a tabletop task, factors might include target x-y cell, yaw, object instance, background, illumination direction and intensity, camera exposure, distractor count and similarity, start joint pose, and occlusion. Freeze valid ranges from the intended operating envelope, then choose combinations that cover important interactions. Changing everything randomly can find failures but makes diagnosis difficult; single-factor sweeps and designed combinations answer different questions.
Keep deterministic regression separate from robustness exploration. A fixed suite with recorded seeds and asset hashes shows whether the same contract changed across builds. A randomized campaign samples a declared distribution and must record every generated scene, including failures. Do not reroll an inconvenient placement or delete a run because the object “looked unfair” after seeing the result. If a condition is invalid, apply a rule written before execution and preserve the excluded record with its reason.
Use an independent oracle and categorize failure at the first unsupported link: target not visible, incorrect detection, pose error, plan failure, grasp slip, transport loss, wrong placement, timeout, unsafe or policy-forbidden action, or evidence loss. Compare success rate with latency and recovery behavior per factor, not only overall. A held-out failure can guide the next training set, but once used for tuning it leaves the untouched test set and becomes development evidence.
Words you need
Name each idea precisely
- Held-out condition
A scene, object, or factor combination reserved from fitting and tuning so it can provide less-biased evaluation evidence.
Physical example:The final test uses taped target cells and lamp directions sealed before model selection and absent from the capstone's training demonstrations.
- Factor
One controlled source of test variation whose influence or interaction the evaluation is designed to observe.
Physical example:Lamp direction is a factor with left, overhead, and right levels while object position and distractor count are separately recorded.
- Distractor
A non-target item or signal that can compete with, resemble, occlude, or redirect the system's perception and action.
Physical example:A second blue block sits near the target cup but should never be grasped or counted as task success.
- Evaluation leakage
Information from reserved evaluation outcomes influencing model, threshold, prompt, scene, or selection decisions before the final claim.
Physical example:After previewing dark-scene failures, an engineer adjusts camera exposure and reports the same scenes as untouched held-out results.
- Scenario seed
A recorded input to pseudorandom scene generation that allows the exact sampled condition to be reconstructed when all other assets and code match.
Physical example:Seed 481 reproduces the same block cell, lamp intensity, distractor pose, and simulated sensor noise under generator version G3.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalReliability, denominators, and fleet rates
Production robotics requires rates, raw counts, time windows, and recoverable releases.
- availability
- healthy service time divided by scheduled timeUnit: percent
- MTTR
- mean time to recoverUnit: minutes or hours
- incidents/robot-hour
- normalized incident rateUnit: 1/hour
Five robots each run 8 h, giving 40 robot-hours.
Two incidents occur, so rate = 2/40 = 0.05 incidents per robot-hour.
Keep raw counts and categories; compare versions under the same window before deciding whether reliability improved.
Use familiar observability and incident response, but join logs to robot ID, physical context, bag replay, and software/model version.
Three incidents across 60 robot-hours gives what rate?
3/60 = 0.05 incidents per robot-hour.
Stratified results are
The combined hides a -percentage-point lighting gap.
Freeze a 24-condition held-out scene matrix
The capstone picks one known object instance from a tabletop. Evaluation varies four reserved position cells, three light directions, and presence or absence of one visually similar distractor.
Define the factors before running: position
{P1,P2,P3,P4}, light{left,overhead,right}, and distractor{absent,present}, giving4 × 3 × 2 = 24labeled combinations.Specify valid physical ranges, fixture marks, lamp placement, camera settings, object and distractor identities, start pose, timeout, reset method, exclusion rule, and independent placement oracle.
Verify from the dataset ledger that reserved position-light combinations were not used for fitting or threshold tuning; disclose any uncertain exposure from generic pretraining.
Generate a signed scenario file for all 24 combinations with fixed order or recorded randomization, unique trial identities, asset hashes, and no option to silently reroll.
Execute in simulation first and then only at the authorized evidence level, recording perception, action, terminal physical result, latency, recovery, and artifact completeness for every condition.
Report results by each factor and interaction, move any condition used for repair into the development set, and create a new untouched holdout for the next final evaluation.
The matrix supplies 24 reproducible, interpretable challenges while keeping prior exposure, invalid-run rules, and evidence level visible.
Unseen testing becomes credible when novelty, range, combination, oracle, and leakage boundary are written before the outcomes appear.
Physical examples
Where this appears in real life
Blue block under a side lamp
A gripper trained mostly with centered blocks under overhead light sees the same object near a workspace edge, with a strong side shadow and a blue tape roll nearby.
Record position, light, shadow, distractor identity, perception result, grasp pose, contact, and final placement so “failed in a harder scene” becomes diagnosable.
Reading a sign beside a similar poster
A child can read a familiar classroom sign until a glossy poster with the same colors reflects sunlight beside it.
Target content stayed constant while glare and visual similarity interacted; varying and naming those factors explains more than simply calling the room unseen.
Hands-on exercise
Make the idea observable
Use the capstone simulator or an approved static tabletop setup with harmless objects and no unreviewed motion. Keep the final 50-trial suite sealed for Day 174.
Inventory training and tuning coverage for object cells, yaw, instances, lighting, background, distractors, start poses, and camera settings; mark unknown exposure honestly.
Reserve at least three factors with named levels inside the intended operating envelope, and place their scenario definitions where model-development code cannot rewrite them casually.
Write success, timeout, forbidden-action, reset, invalid-trial, and artifact-completeness rules before observing evaluation output.
Run a small deterministic matrix with immutable scenario identities, then replay one seed to verify the scene and initial state truly reconstruct.
Produce per-factor and interaction counts alongside overall results, and classify each failure at the earliest evidenced boundary without guessing a root cause.
Choose one failure for future repair, copy it into a development regression set, and explicitly retire it from untouched held-out status.
A single overall score hides edge-position and side-light interactions, while the exposure ledger often reveals that several supposedly unseen scenes influenced earlier tuning.
Every run can be reconstructed, no failed scene was rerolled, the report states what was genuinely held out, and tuned-on failures are no longer presented as untouched evidence.
Build today
Build a SIL→HIL→hardware CI ladder with containers, diagnostics, fleet telemetry, rollback, failure replay, and a frozen 50-trial acceptance suite.
Evidence to save
DONE when a comparison table for “Test unseen object positions, lighting, and distractors” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Calling any newly recorded scene unseen after inspecting and tuning against its failures.
Track exposure and decision use; once a scene guides model, threshold, prompt, or configuration changes, move it to development and reserve new untouched evidence.
Changing many factors randomly and reporting only one overall success percentage.
Record every factor and seed, include controlled sweeps or designed interactions, and publish stratified counts so a failure pattern remains visible.
Deleting a failed run because the placement looked unreasonable afterward.
Apply only the predeclared validity rule, preserve the trial and exclusion reason, and revise the next protocol rather than editing the current denominator.
Job connection
How this becomes employable evidence
Design held-out robot-policy evaluations across workspace, illumination, clutter, object, and start-state factors, protect the leakage boundary, preserve deterministic regressions, and turn physical failures into correctly scoped data work.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 25 interview drill
Interview questions: Test unseen object positions, lighting, and distractors
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Your policy scores well on familiar scenes but fails near table edges under side light. Design a held-out matrix, leakage controls, oracle, factor report, and rule for converting failures into training data.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1When does a held-out scene stop being held out?
When its inputs or outcome influence model, data, threshold, prompt, configuration, or selection decisions for the claimed final evaluation.
Q2Why record both a seed and generator version?
The same seed can produce a different scene when generator logic or assets change, so both identities are needed for reconstruction.
Q3Why report factor interactions instead of only individual factors?
The policy may tolerate edge position and side light separately but fail when their effects occur together, which marginal totals can hide.