Chapter 25 · Operate the capstone as a reproducible, diagnosable robot service
Today in the field story
One problem, then the next
Release Train Fifty now runs exactly the declared fifty rows. Every attempt records configuration, stage, start, terminal outcome, intervention, prohibited event, timing, recovery, artifacts, and adjudication. No invisible retry replaces a failed row; invalid tests follow the predeclared policy; safety-critical failures remain blocking rather than averaged into a percentage. Counts reconcile with raw records before any summary appears. The train may end in release, revise, or block, but it cannot rewrite thresholds to manufacture the desired station arrival.
- Why now
Frozen repeated trials convert a demonstration into bounded reliability evidence.
- Ignore today
Do not rerun failures selectively or change denominators and gates.
- Unlocks next
A complete acceptance ledger and defensible decision.
Understand
Build the physical picture first
A frozen 50-trial suite is a sealed box of numbered challenge cards: rules, order, thresholds, resets, and failure gates cannot change after the first card is opened.
Freeze the protocol before execution: intended task and environment, build and configuration identities, 50 scenario identities, order or randomization rule, operator actions, initial state, reset, timeout, success oracle, failure categories, invalid-trial rule, artifacts, and acceptance thresholds. The number 50 comes from this capstone gate, not a universal law of reliability. It provides a bounded comparison for this declared suite; it does not prove every future hour, object, operator, or environment will behave the same.
Define one atomic trial from verified ready state to independently measured terminal state. Record every attempt, including setup errors and infrastructure failures, under a predeclared classification. Do not pause the denominator, rerun a miss, adjust a pose, or switch models after seeing performance. If a required camera, log, or oracle is missing, the trial may be invalid under the frozen rule, but the missing evidence still remains visible and may block acceptance if the allowed invalid count is exceeded.
Use compound acceptance, not one average. A hypothetical plan might require at least 47/50 task successes, zero forbidden-object grasps, zero unplanned contacts, zero stale-command violations, all cancellation deadlines met, and complete evidence for at least 49 trials. The exact thresholds must come from the project's requirements and risk review. One high-consequence violation cannot be washed away by 49 easy successes, and rounding 46/50 = 92% to a preferred target is dishonest.
Summarize counts by frozen factor, failure mode, time, and recovery, then preserve uncertainty. Fifty observations estimate performance only under sampled conditions; dependent trials, changing temperature, object wear, and operator resets can reduce effective independence. Acceptance is a decision against a protocol, not proof of a universal probability. If the suite fails, retain it, issue a disposition, repair outside the frozen run, and execute a new versioned suite rather than splicing replacement trials into history.
Words you need
Name each idea precisely
- Frozen protocol
A versioned test definition whose scenarios, rules, thresholds, and evidence requirements are locked before outcomes are revealed.
Physical example:A signed manifest lists trials T001–T050, their fixture marks, order, reset, timeout, oracle, artifacts, and decision gates before T001 begins.
- Atomic trial
One indivisible attempt from declared initial state through terminal outcome, counted without replacing an inconvenient middle segment.
Physical example:The arm starts at home, receives one pick request, either places the block by 20 seconds or records a failure, then follows the fixed reset.
- Critical gate
A zero-tolerance or separately bounded condition whose violation determines rejection regardless of aggregate success count.
Physical example:Grasping the red forbidden object once rejects the run even when all other 49 target placements succeed.
- Denominator
The full set of attempts governed by the counting rule, including declared treatment of invalid, aborted, or infrastructure-affected trials.
Physical example:Removing two timeouts makes 46 successes look like
46/48, while the frozen report correctly retains all 50 scheduled attempts and their statuses.- Acceptance disposition
The explicit
accept,conditional, orrejectdecision tied to frozen gates, deviations, evidence, owner, and scope.Physical example:A 46-success result against a 47-success gate is
reject, with four preserved trial IDs and a new-suite requirement after repair.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalReliability, denominators, and fleet rates
Production robotics requires rates, raw counts, time windows, and recoverable releases.
- availability
- healthy service time divided by scheduled timeUnit: percent
- MTTR
- mean time to recoverUnit: minutes or hours
- incidents/robot-hour
- normalized incident rateUnit: 1/hour
Five robots each run 8 h, giving 40 robot-hours.
Two incidents occur, so rate = 2/40 = 0.05 incidents per robot-hour.
Keep raw counts and categories; compare versions under the same window before deciding whether reliability improved.
Use familiar observability and incident response, but join logs to robot ID, physical context, bag replay, and software/model version.
Three incidents across 60 robot-hours gives what rate?
3/60 = 0.05 incidents per robot-hour.
For successes in frozen trials,
The remaining four trials stay in the ledger; hidden reruns cannot replace failures outside a predefined invalid-trial rule.
Adjudicate a hypothetical 50-trial capstone run
Protocol ACC-25-v1 requires at least 47 successful placements, zero forbidden grasps, zero unplanned contacts, cancellation within its declared deadline, and at least 49 complete evidence bundles.
Verify before T001 that source, image, model, resolved configuration, calibration, fixture, object set, scenario list, order, operator, clocks, oracle, safety authorization, and output storage match the signed protocol.
Execute T001–T050 exactly once each, record start and terminal states plus artifacts immediately, and use only the frozen invalid-trial and abort rules.
Count a hypothetical result of 46 successes, two grasp slips, one timeout, and one wrong-placement failure; all 50 have evidence, no forbidden grasp or unplanned contact occurs, and cancellation deadlines pass.
Calculate task success as
46 / 50 = 0.92 = 92%; keep failure identities and factor strata visible instead of rounding, excluding, or replacing the four unsuccessful trials.Compare every gate: evidence, forbidden action, contact, and cancellation pass, but
46 < 47, so the compound protocol fails even though several critical gates stayed clean.Issue
REJECT for ACC-25-v1, preserve the complete run, open hypotheses for the four failures, and require a separately versioned post-fix suite rather than adding one replacement success.
The hypothetical run is rejected at 92% because it misses its frozen 47-success threshold; its clean critical gates remain useful evidence but do not rewrite the decision.
Acceptance is mechanical only after thoughtful rules are frozen: count every governed attempt, evaluate each gate, and resist negotiating with visible results.
Physical examples
Where this appears in real life
Fifty basketball free throws
A player takes 50 shots from taped spots with the same ball, counted in a sealed order; moving closer after misses or ignoring a camera outage changes the test.
Fixed position, order, counting, and independent observation make the result comparable, but it still says little about a different ball, court, fatigue pattern, or defender.
Forbidden red block among targets
A tabletop suite contains blue targets and a red distractor. The robot completes 49 placements but grasps the red object once.
If wrong-object grasp is a frozen critical gate, the correct disposition is reject; 98% success cannot neutralize the specific unacceptable behavior.
Hands-on exercise
Make the idea observable
Use the frozen capstone suite in simulation, HIL, or an approved controlled-hardware area. If only SIL is available, run all 50 there and label the result SIL-only without extending the claim.
Seal a machine-readable manifest for exactly 50 uniquely identified scenarios, their order, build, model, configuration, environment, resets, timeouts, oracle, artifacts, gates, and abort rules.
Run a preflight that checks authorized evidence level, fixture, devices, calibration, clocks, storage, operator, safeguards, and no-motion recovery before starting T001.
Execute T001 through T050 sequentially without model or threshold changes, hidden retries, deleted failures, or ad hoc scene adjustments; preserve each terminal record immediately.
Stop only under the frozen abort rule, keep all completed and aborted records, and never resume after a configuration change under the same suite identity.
Calculate exact counts and rates for task result, factor strata, failure categories, deadlines, forbidden behaviors, contacts, recoveries, invalid trials, and artifact completeness.
Apply each gate independently, write the acceptance disposition and scope, and have a second reviewer reconcile the manifest, 50 identities, artifact checksums, counts, and decision.
If rejected, move failures into a separate development queue, retain the sealed run untouched, and prepare a new protocol version only after corrective work and review.
Execution discipline reveals tempting shortcuts: retrying a near miss, improving lighting mid-run, omitting an awkward failure, or calling a SIL result hardware acceptance.
Exactly 50 governed trial records reconcile with the frozen manifest, every gate has a decidable result, failures remain present, and the acceptance claim names its precise evidence level and conditions.
Build today
Build a SIL→HIL→hardware CI ladder with containers, diagnostics, fleet telemetry, rollback, failure replay, and a frozen 50-trial acceptance suite.
Evidence to save
DONE when a 60–120 second uncut “Run the frozen 50-trial acceptance suite” demo links to its command, logs or plots, result count, and honest failure note.
Common mistakes
Catch the wrong mental model
Rerunning failed trials and keeping only the best attempt.
Count the original governed attempt, preserve its evidence, and place any diagnostic or post-fix rerun in a separately labeled dataset or suite.
Changing an acceptance threshold because the observed result was close.
Apply the frozen gate, record rejection or deviation through the authorized process, and revise only a future protocol without rewriting this run.
Presenting 50 successes as proof of universal reliability.
Limit the claim to the sampled conditions, identities, resets, time window, dependence structure, oracle, and evidence level; publish unsampled risks and uncertainty.
Job connection
How this becomes employable evidence
Own a customer-facing robot acceptance campaign, lock configurations and scenario strata, run sequential trials under approved controls, reconcile independent physical evidence, enforce critical gates, and defend a reject decision against metric shopping.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Application / ROS 2 Integration Engineer
Chapter 25 interview drill
Interview questions: Run the frozen 50-trial acceptance suite
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Your capstone gets 46 successes in 50 against a frozen 47-success gate with no safety-critical event. Decide the result, explain the denominator and evidence scope, and describe the valid post-fix process.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Can one failed trial be replaced when the reset looked imperfect?
Only if the frozen invalid-trial rule clearly applies; preserve the original record and any replacement identity, and follow the predeclared denominator rule.
Q2Why can 49 ordinary successes not cancel one critical-gate failure?
The critical behavior has an independent acceptance limit based on its consequence, so aggregate task success is the wrong decision rule.
Q3What should happen after a rejected suite leads to a code fix?
Keep the rejected suite unchanged, add the case to development regression, and execute a new versioned acceptance suite under newly frozen identities.