Chapter 17 · Prove FleetOps across regression, interfaces, safety, and commissioning

Today in the field story

One problem, then the next

Place each R-17 check at the smallest boundary capable of exposing its defect: pure transition logic, service integration, ROS 2 launch, or complete mission. Plant a passing unit test beside a failing launch interaction to show why test count is not confidence. Preserve logs and terminal state for every layer, and keep one end-to-end path that proves the pieces agree under cancellation and restart.

Why now: Traceable requirements now need tests chosen for fault-detection power, not convenience.
Ignore today: Ignore maximizing coverage percentages; justify each layer and its blind spots.
Unlocks next: A regression stack that can feed deterministic scenarios and CI.

Understand

Build the physical picture first

Regression confidence is a layered net: each test level must cross a real boundary and catch a named defect that cheaper, narrower checks intentionally cannot see.

Use pytest for Python logic and data contracts, and gtest for C++ algorithms, state machines, boundary arithmetic, and class behavior. Keep these component tests fast and explicit, but do not pretend an in-process fake proves middleware discovery, process lifecycle, configuration loading, clocks, executor behavior, or a physical result. A useful test double narrows one dependency; a hidden fallback that makes the product behave differently under test invalidates the evidence.

An integration test connects real production interfaces between selected components: serialized messages, database constraints, ROS services or actions, adapters, and cancellation acknowledgments. ROS 2 launch_testing can launch processes, run active tests while they are alive, and run post-shutdown checks over exit codes, output, and generated evidence. Waiting assertions require timeouts, and every process match must be specific enough that output from the wrong node cannot accidentally satisfy the test.

A system test begins at an external mission request and ends at the declared physical or simulated outcome. It should inspect intermediate state only to diagnose the route, not redefine success as “HTTP 200,” “action accepted,” or “process exited zero.” Acceptance depends on the terminal mission, measured robot state, safety or fault state, and correlated artifacts. Plant schema mismatch, process death, delayed acknowledgment, and wrong final position so the expected layer actually demonstrates its detection power.

Regression selection follows change and risk, not a fixed pyramid slogan. Run the narrow checks that localize arithmetic and contracts, then the smallest cross-process set covering touched interfaces, then risk-linked mission scenarios and required SIL or HIL gates. Quarantine is not a trash bin for inconvenient failures: a flaky test needs an owner, suspected mechanism, evidence, impact assessment, and deadline, while a safety- or acceptance-critical flake blocks the claim it was meant to support.

Words you need

Name each idea precisely

Component test: A narrow executable check of one algorithm, class, function, or data contract with controlled dependencies.
Physical example:
A gtest case proves a mission state machine rejects completed directly after queued and reports the invalid transition.
Integration test: A check that selected production components communicate through their real contract and failure semantics.
Physical example:
A Python mission client sends a versioned request through the real adapter serializer and verifies cancellation acknowledgment plus stored terminal state.
launch_testing: The ROS 2 framework for launching processes and applying active and post-shutdown assertions to their runtime behavior and results.
Physical example:
The test launches allocator and adapter nodes, waits for one specifically matched readiness event, kills the adapter, then checks bounded fault publication and exit evidence.
System test: An end-to-end evaluation of the integrated product against an external requirement and terminal outcome in a declared environment.
Physical example:
A request for tote delivery is accepted once, assigned, executed in simulation, acknowledged by the station emulator, and closed with a measured final pose.
Test oracle: The independent rule or measurement that decides whether observed behavior is correct.
Physical example:
The oracle checks the simulated robot pose and station handshake, not the fleet service’s own optimistic completed field.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalRisk priority, trial denominators, and recovery time

Validation turns hazards into traceable tests and reports every planned trial, including the failures that make a result uncomfortable.

RPN = S×O×D: an ordinal FMEA priority from severity, occurrence, and detection ratingsUnit: relative score
p̂ = k/N: observed passes k divided by all planned trials NUnit: fraction or percent
Tᵣ: time from a declared failure event until every recovery condition remains trueUnit: seconds (s)

A hazard is rated severity S=5, occurrence O=2, and detection difficulty D=4, so its relative RPN is 5×2×4 = 40.
A frozen matrix planned N=20 trials and passed k=17, so the observed pass rate is 17/20 = 85%; the three failures stay in the denominator.
If valid sensing returns at 12.0 s and all stability criteria hold from 15.5 s onward, report recovery time Tᵣ = 3.5 s and preserve the trace.

Programmer analogy

Treat it like a release test matrix with trace IDs, except the failed requirement can concern motion, collision, or loss of control rather than a screen defect.

A frozen suite passes 27 of 30 planned trials. What observed pass rate must be reported?

27/30 = 0.90 = 90%, with all three failures retained and categorized.

Total test count is

N=600+60+12=672.

Component tests alone are

\frac{600}{672}\times100\%\approx89.3\%

of cases but include $0$ of the $12$ full-system missions.

Choose the first test level that can expose four defects

A FleetOps suite has 480 component checks taking 24 seconds, 40 cross-process integration checks taking 5 minutes, and 8 complete simulated missions taking 32 minutes.

Calculate the case total as 480 + 40 + 8 = 528; component checks are 480 / 528 ≈ 90.9% of case count but cover zero complete missions.
Assign a wrong battery-threshold comparison to a component test because one pure function can reproduce and localize the arithmetic defect.
Assign a renamed serialized field between allocator and adapter to an integration test using the real schema and deserializer rather than duplicated in-test structures.
Assign a ROS adapter process that dies after startup to launch_testing so an active timeout and post-shutdown exit assertion inspect the launched system.
Assign “service reports complete while the robot stops at the wrong station” to the full mission test with an independent terminal-pose and station-handshake oracle.
Order the regression run by fast localization first, then touched interfaces, then all risk-linked complete scenarios; retain each planted failure as proof that its assigned layer detects it.

Result

The suite uses all 528 checks but never converts the 90.9% component-test share into a false claim of end-to-end or acceptance coverage.

What this proves

A test earns confidence from the production boundary and failure it challenges, not from its runner, speed, or contribution to a large count.

Physical examples

Where this appears in real life

Airport checks at different boundaries

A paper itinerary separates passport validation, airline-to-border data exchange, and the traveler’s complete arrival at the correct gate with luggage.

Look for:

The document scanner can pass alone while an interface or full journey fails; counting the fastest checks cannot substitute for traversing the missing boundary.

Lamp circuit from part to outcome

With a battery-powered classroom lamp disconnected from mains, check the switch, then the switch-to-wire connection, then whether pressing the assembled switch actually illuminates the bulb.

Look for:

Part correctness, connection correctness, and observable product outcome answer different questions even in a very small system.

Hands-on exercise

Make the idea observable

Use the simulated FleetOps workspace and its actual package interfaces. If a required node, toolchain, or simulator is unavailable, record that exact boundary as blocked rather than replacing it with a hidden mock.

Add one pytest case for mission-request schema and idempotency validation, including a malformed field and a duplicate correlation key.
Add one gtest or inspectable C++ test design for a bounded mission-state transition, including an illegal transition and the expected diagnostic.
Create one launch_testing case that starts at least two real ROS 2 processes, waits with a finite timeout, injects one process termination, and verifies the specifically matched fault plus shutdown result.
Run one complete simulated mission from external API request to independently measured terminal pose and station acknowledgment; retain a single trace identity across all layers.
Plant schema mismatch, timeout, process death, and false-completion defects one at a time, then record the first layer that detects each and every layer that legitimately cannot.
Publish command, duration, exact versions, pass and fail counts, raw result locations, planted-defect evidence, and any unexecuted HIL or physical boundary without inflating the claim.

Observe

Fast tests localize code defects, launch tests expose runtime and lifecycle behavior, and the complete mission reveals outcomes that no internal status field can prove by itself.

Done when

All four planted defects are caught at an intentional layer, the nominal mission reaches an independently verified terminal outcome, and no mock-only result is labelled system acceptance.

Build today

Create a risk-linked SIL→HIL acceptance ladder for FleetOps, automate regression scenarios, integrate one external fleet or PLC boundary, and publish FAT/SAT evidence plus an incident report.

Evidence to save

DONE when “pytest, gtest, launch_testing, and component-to-system regression” runs from one documented command and the nominal plus boundary outputs are attached.

Common mistakes

Catch the wrong mental model

Wrong

Calling a launched process test end to end because several nodes started.

Better

Name the actual boundary crossed and require an external request plus independent terminal outcome before claiming a complete mission.

Wrong

Waiting forever for a ROS message or matching any process output containing a convenient word.

Better

Use bounded waits and specific process or action identities so a hang fails and unrelated output cannot satisfy the assertion.

Wrong

Quarantining a flaky acceptance test while continuing to claim the risk is covered.

Better

Assign an owner and mechanism investigation, preserve rerun evidence, and mark the associated acceptance claim blocked until a trustworthy check or equivalent proof exists.

Job connection

How this becomes employable evidence

Design a ROS 2 regression lane that combines Python contracts, C++ state-machine checks, launch_testing process faults, and independently measured simulated missions, while keeping commands, artifacts, timing, and uncovered hardware boundaries explicit.

Relevant target roles

Robotics Deployment, Integration & Validation Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Application / ROS 2 Integration Engineer

Chapter 17 interview drill

Interview questions: pytest, gtest, launch_testing, and component-to-system regression

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Your team has thousands of passing unit tests but field missions still fail. Explain what pytest, gtest, launch_testing, integration, and system tests each prove, then place schema mismatch, process death, timeout, and wrong physical outcome.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What can a launch_testing post-shutdown test inspect that an active test normally cannot yet know?

Model interview answer

It can inspect final exit codes, complete output, and artifacts after the launched processes have terminated.

Q2Why is an accepted mission response not a system-test oracle?

Model interview answer

It proves only that one service accepted or recorded a request; it does not prove allocation, execution, physical or simulated outcome, station interaction, or safe terminal state.

Q3How should regression depth change with risk?

Model interview answer

Run narrow localization checks plus every real interface, mission, SIL, HIL, and acceptance case needed by the touched requirements and failure consequences.