Chapter 25

Operate the capstone as a reproducible, diagnosable robot service

Turn the Field Capstone from a successful policy demonstration into an inspectable production candidate. This chapter builds a risk-linked SIL→HIL→hardware acceptance ladder; packages software without pretending a container includes firmware, devices, or calibration; provisions hosts and validates versioned configuration; captures diagnostics, metrics, traces, core dumps, and replay inputs with one evidence identity; designs truthful fleet telemetry, idempotent mission APIs, signed staged updates, and tested rollback; challenges held-out object positions, lighting, and distractors; freezes and executes exactly 50 acceptance trials; and publishes an uncut demonstration plus a failure analysis that preserves weak results. Every pass is limited to the tested build, configuration, hardware, environment, and oracle.

Before you start

Bring the Week 24 capstone's frozen model, export artifact, measured edge profile, ROS 2 action boundary, held-out evaluation, fallback behavior, and exact code, model, configuration, and dataset identities.
Bring one safe simulation baseline and, if available, an approved restricted-energy HIL bench or robot test area. A simulator, controller board, and complete robot are different evidence levels and must never share one unlabeled pass.
Be able to distinguish request, acceptance, command, measured robot state, physical task result, and operator-visible terminal state; retain one mission identity across those boundaries.
Keep learner work in simulation or on an authorized bench. Real motion, intentional communication loss, power interruption, watchdog tests, firmware updates, recovery, and rollback require the manufacturer's procedure, qualified supervision, a controlled area, and the site's safety and change controls.

By the end

Build a risk-linked CI matrix that states what SIL, HIL, and controlled-hardware stages contain, which defects each can expose, what blocks promotion, and exactly what each passing stage does not prove.
Create a deployment manifest joining immutable container digest, application build, model, host image, driver and firmware versions, device identities, configuration schema and content hashes, calibration, and rollback compatibility.
Correlate diagnostics, structured logs, metrics, ROS 2 traces, core dumps, mission events, and recorded inputs into an access-controlled incident bundle, then replay it without claiming replay equals the original physical event.
Design fleet telemetry around freshness and reported state, a mission API around idempotency and legal transitions, and a secure OTA campaign around device identity, signed metadata, compatibility, canaries, health gates, and recovery.
Freeze a held-out scenario matrix over object position, lighting, and distractors; prevent evaluation leakage; and separate deterministic regression cases from randomized robustness exploration.
Execute and adjudicate a frozen 50-trial suite without retries, threshold changes, hidden exclusions, or safety-critical failures being averaged away.
Publish an uncut run, artifact manifest, acceptance table, and fact-first failure analysis whose claims remain bounded to the recorded configuration and whose failed evidence is never overwritten.

The field story

Release Train Fifty

Policy Forge Model v1 and Dataset v1 board Release Train Fifty with exact hashes, fallback behavior, and known limitations. Your mission is to determine whether that frozen capstone can operate as a reproducible, diagnosable service. SIL, HIL, and controlled hardware remain distinct stations, each with named contents, defects it can expose, promotion gates, and explicit non-claims. A container cannot smuggle in firmware or calibration identity, and a passing simulator cannot become hardware evidence through optimistic wording. Every mission keeps one correlation identity from request to observed task outcome.

The train then builds its operational spine: immutable deployment manifests, versioned configuration, structured diagnostics, traces, dumps, and replay inputs. Fleet telemetry reports freshness rather than pretending to know physical truth; mission APIs enforce identity and legal transitions; update campaigns require signed metadata, compatibility, canaries, health gates, and rollback. Unseen conditions remain frozen before execution. Exactly fifty declared trials run without hidden retries or threshold changes, and the final uncut demonstration indexes the full evidence rather than replacing it. Weak results stay published beside the scoped release, revise, or block decision.

Why this chapter now: Week 24 froze a candidate bundle. Production credibility now depends on deployment identity, staged evidence, observability, rollback, and fixed acceptance trials rather than another model demonstration.
Ignore for now: Do not merge SIL, HIL, and hardware claims, hide retries, change thresholds after results, equate replay with the physical incident, or perform unauthorized hardware and update faults.
This unlocks: A capstone evidence package strong enough for Week 26 portfolio stories, incident interviews, role matching, and honest statements about what remains unverified.
Proof you will leave with: A stage matrix, deployment/configuration manifest, correlated telemetry and incident bundle, replay result, mission and OTA state contracts, rollback evidence, frozen robustness manifest, 50 raw trial rows, uncut demo, and fact-first failure analysis.

Environment contractrepository-supported Node.js 22.13.0 or newer and deterministic synthetic station results support the starter; SIL is the default learner evidence, while HIL and controlled hardware require separately available authorized environments.

Compatibility boundary: The starter is not a CI, container, OTA, fleet, or hardware implementation. Those paths must pin the actual host, image, driver, firmware, devices, calibration, security metadata, robot, network, and rollback compatibility.
Smoke check: Run node week-25-release-train-fifty.mjs; confirm SIL passes, the planted missing rollback evidence blocks HIL promotion, and hardware remains unverified.
Contract reviewed: 2026-07-25
Runtime evidence: The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
Drift risk: high

Today in the field story

One problem, then the next

Release Train Fifty first assigns every risk to the earliest useful station. SIL checks deterministic logic, interfaces, faults, and orchestration without hardware. HIL may expose device timing and controller integration on an approved bench. Controlled hardware adds physical behavior inside a defined area. Each pass states what was present and what remains unproved. The planted HIL rollback gap demonstrates that functional success alone cannot promote a candidate whose recovery path has not survived the same stage.

Why now: Stage labels must preserve evidence level and promotion risk.
Ignore today: Do not simulate unavailable HIL or hardware passes.
Unlocks next: A risk-linked ladder with fail-closed promotion rules.

Understand

Build the physical picture first

The acceptance ladder is three differently equipped rooms: software rehearsal, controller bench, and controlled robot cell, with a locked evidence door between each room.

Picture one grasp command travelling toward a real arm. In software-in-the-loop, or SIL, the production decision software talks to a simulated robot and simulated sensors. In hardware-in-the-loop, or HIL, at least one production hardware boundary—such as the target computer, controller, field bus, or sensor interface—participates while dangerous motion is absent or tightly controlled. The hardware stage exercises the integrated robot in its approved test area. Passing one room grants entry to the next; it does not repaint that earlier evidence as physical proof.

Freeze an acceptance contract before connecting it to CI. Each row needs a requirement or risk identifier, test level, exact stimulus, independent oracle, threshold, timeout, required artifacts, runner identity, and blocking rule. An oracle is how the test knows the truth; the policy node reporting success is not independent if that same node is under test. CI contributes repeatable triggering and record keeping, but a green badge alone says nothing about sensor realism, physical contact, calibration, or protective functions.

Place a test at the cheapest level that can expose its target defect, then repeat high-consequence contracts across boundaries. SIL can rapidly explore state-machine transitions, deterministic time jumps, and thousands of perception inputs. HIL can reveal target-architecture, driver, bus timing, device startup, and watchdog defects that simulation omits. Controlled hardware can reveal backlash, glare, cable drag, thermal drift, object compliance, and real stopping behavior. Safety validation remains a separate qualified activity; a CI job must not bypass interlocks or create unapproved motion.

Promotion depends on immutable evidence, not an impressive percentage. Record source revision, binary and model digests, merged configuration hash, scenario and seed, simulator or bench version, hardware serial or fixture identity, calibration, start time, monotonic timestamps, result, and artifact checksums. A flaky required case is unresolved evidence, not a pass after enough retries. If HIL is unavailable, the honest result is SIL passed; HIL and hardware not verified, with the missing boundary and owner named.

Words you need

Name each idea precisely

Software-in-the-loop (SIL): Production-intent software tested against simulated plant, sensors, time, or peer interfaces without claiming the corresponding physical hardware behaved.
Physical example:
A simulated camera publishes a shifted block pose and a simulated arm reports whether the policy's bounded grasp reaches the virtual target.
Hardware-in-the-loop (HIL): A test that includes a named real controller, compute target, device, or electrical interface while the remaining plant is simulated or safely emulated.
Physical example:
The production edge computer sends commands through the real CAN interface to a load-free controller bench that returns scripted encoder frames.
Test oracle: An observation or rule, sufficiently independent of the implementation under test, that decides whether the required outcome occurred.
Physical example:
An overhead camera and fixture switch judge successful placement instead of trusting the policy's own done=true message.
Promotion gate: A predeclared set of blocking results and evidence requirements that must be satisfied before a build enters a higher-risk test level.
Physical example:
The controller bench stays disabled until all SIL cancellation cases pass on the same signed build and configuration digest.
Evidence identity: The immutable set of build, configuration, environment, runner, hardware, scenario, and artifact identifiers attached to one result.
Physical example:
Trial HIL-018 names commit, image digest, model hash, controller serial, firmware, configuration hash, fault script, and trace checksum.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalReliability, denominators, and fleet rates

Production robotics requires rates, raw counts, time windows, and recoverable releases.

availability: healthy service time divided by scheduled timeUnit: percent
MTTR: mean time to recoverUnit: minutes or hours
incidents/robot-hour: normalized incident rateUnit: 1/hour

Five robots each run 8 h, giving 40 robot-hours.
Two incidents occur, so rate = 2/40 = 0.05 incidents per robot-hour.
Keep raw counts and categories; compare versions under the same window before deciding whether reliability improved.

Programmer analogy

Use familiar observability and incident response, but join logs to robot ID, physical context, bag replay, and software/model version.

Three incidents across 60 robot-hours gives what rate?

3/60 = 0.05 incidents per robot-hour.

Failure counts at each gate are

N_{\text{SIL fail}}=1{,}000(1-0.995)=5,

N_{\text{HIL fail}}=100(1-0.98)=2,\qquad N_{\text{hardware fail}}=20(1-0.95)=1.

Turn one stale-pose risk into a three-level promotion row

The capstone must cancel a grasp when camera-pose age exceeds 250 ms. The software response deadline is 150 ms after that threshold; the physical terminal condition is governed by a separate approved procedure.

Create REQ-VISION-07 and state the trigger, pose-age clock, 250 ms freshness limit, 150 ms software-response deadline, expected action status, and the physical claim that this software-only requirement does not decide.
Define SIL-07 to freeze the simulated source timestamp while messages continue, using an external probe to time first threshold crossing, cancellation request, action terminal state, and absence of later motion commands.
Define HIL-07 with the identical logical fault injected through the target edge computer and production camera interface emulator; add device startup, driver, and bus evidence that SIL cannot supply.
Define HW-07 only by reference to the reviewed robot procedure, controlled area, qualified owners, independent measurement, allowed speed and payload, stop criteria, and abort conditions; do not invent those values in CI.
Freeze promotion rules: SIL and HIL require the same build and merged-configuration hash, zero missing artifacts, no retries, and all timing assertions met; hardware scheduling remains blocked until the responsible reviewers approve.
If SIL measures 118 ms and HIL measures 164 ms, record SIL PASS, HIL FAIL, hardware NOT RUN, retain both traces, and investigate target scheduling instead of averaging 141 ms into a pass.

Result

One requirement produces comparable evidence at two software boundaries and a separately governed physical test, while the 164 ms HIL failure correctly blocks higher-risk execution.

What this proves

A ladder is trustworthy when each rung names its equipment and oracle, carries forward the same contract, and refuses to borrow confidence from a different boundary.

Physical examples

Where this appears in real life

Three-room marble sorter

A virtual sorter first routes drawn marbles on a screen, then a real controller switches small indicator lamps from simulated sensors, and only afterward an enclosed physical sorter handles foam balls.

Look for:

The same routing contract crosses three boundaries, while lamp timing cannot prove ball handling and a successful virtual route cannot prove wiring or mechanics.

Frozen encoder at successive gates

SIL freezes a simulated joint timestamp, HIL replays the same stale frame through the target interface, and a qualified hardware procedure challenges the approved sensor path without defeating safeguards.

Look for:

Detection deadline and terminal application state can be compared across levels, but only the final authorized setup observes the actual mechanism and its physical response.

Hands-on exercise

Make the idea observable

Use the existing capstone simulator, a CI configuration draft, and a paper HIL/hardware plan. Do not connect, energize, move, fault, or update physical equipment for this exercise.

List eight capstone risks or requirements and mark the first test level able to observe each defect: component, SIL, HIL, or controlled hardware.
Choose three high-consequence contracts that must repeat at more than one level, and keep their triggers, units, terminal states, and evidence names consistent.
Write a matrix with build, model, configuration, scenario, seed, runner, device, calibration, oracle, timeout, artifact, result, and promotion columns.
Automate one safe SIL nominal case and one injected failure with deterministic setup, bounded timeout, clean cancellation, and a machine-readable result.
Draft HIL and hardware rows using NOT AVAILABLE or NOT AUTHORIZED where appropriate; name the exact missing equipment, procedure, reviewer, or measurement instead of simulating a pass.
Have another engineer trace one green row from requirement through command, measured outcome, artifacts, and promotion decision, then repair every unsupported link.

Observe

The exercise usually exposes tests whose oracle is the component under test, hardware claims supported only by simulation, and results that cannot be reconstructed from a build or configuration identity.

Done when

The SIL job is repeatable, every higher-level row states its real boundary and authority, failures block promotion, and the report contains no HIL or hardware claim without matching evidence.

Build today

Build a SIL→HIL→hardware CI ladder with containers, diagnostics, fleet telemetry, rollback, failure replay, and a frozen 50-trial acceptance suite.

Evidence to save

DONE when the learning log explains “Build a SIL→HIL→hardware CI acceptance ladder” in five precise points and a checked example produces the predicted output.

Common mistakes

Catch the wrong mental model

Wrong

Calling a target computer running against a fully simulated device a real-robot test.

Better

Label the participating boundary precisely as HIL, list the simulated plant and interfaces, and reserve hardware acceptance for the controlled integrated robot and independent physical measurements.

Wrong

Retrying a required flaky case until CI becomes green.

Better

Keep the original failure, mark the evidence unresolved, diagnose the flake, and rerun only under the predeclared policy without converting repeated attempts into one clean result.

Wrong

Using the policy's own success field as the sole acceptance oracle.

Better

Observe the terminal task independently through measured robot or fixture state and correlate that observation with the internal action result for diagnosis.

Job connection

How this becomes employable evidence

Design a risk-based regression pipeline for a robot release, place faults at SIL, target-compute HIL, and controlled-hardware boundaries, preserve traceable artifacts, and stop promotion when timing or evidence is incomplete.

Relevant target roles

Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 25 interview drill

Interview questions: Build a SIL→HIL→hardware CI acceptance ladder

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Explain how you would test stale perception through SIL, HIL, and real hardware. Name the equipment, oracle, promotion rule, evidence limitation, and what a green CI job still cannot prove.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What does a SIL pass prove about a physical gripper?

Model interview answer

Only that the named software behaved against the declared simulation and oracle; it does not prove wiring, firmware, timing, force, contact, calibration, or physical task success.

Q2When should a test appear on more than one ladder rung?

Model interview answer

When the same high-consequence contract needs fast regression plus evidence from target hardware or physical effects that a cheaper level cannot observe.

Q3What is the correct status when SIL passes but no HIL bench exists?

Model interview answer

SIL passed; HIL and hardware not verified, followed by the missing boundary, required evidence, owner, and promotion block.

Chapter references

Docker Docs — Building best practicesOfficial guidance for small trusted base images, multi-stage builds, cache behavior, CI builds, mutable tags, and digest pinning; used to teach image identity without claiming that a container captures host, device, firmware, calibration, or secret state.
ROS 2 — ros2_tracingROS 2 project source for trace instrumentation, userspace and optional kernel tracing boundaries, launch and command control, initialization metadata, and snapshot capture; learners must measure overhead and retention for their own target.
Open-RMF — Fleet Adapter TutorialMaintainer tutorial showing a fleet adapter as the bridge between robot state, task response, vendor robot commands, configuration, navigation graph, and RMF scheduling; its example REST interface is not treated as a universal production API.
Uptane Standard 2.1.0Primary secure-update standard for signed metadata, freshness, hardware targeting, key-compromise resilience, and rollback or freeze attack defenses; used as architecture evidence, not as a claim that a classroom updater is Uptane-conformant.
NIST — Grasping, manipulation, and contact-safety performanceGovernment measurement program illustrating declared metrics, test methods, artifacts, independent measurement, repeatability, and recorded assembly-test configuration; the capstone suite remains project-specific rather than a NIST certification.