Chapter 17
Prove FleetOps across regression, interfaces, safety, and commissioning
Turn the simulated FleetOps build into a risk-linked acceptance package rather than a convincing demonstration. This chapter moves from testable requirements and failure analysis through component, ROS 2 launch, system, deterministic simulation, SIL, and HIL evidence; challenges external fleet and industrial boundaries; keeps functional safety under qualified ownership; and closes with FAT, SAT, commissioning, incident, rollback, and explicit go/no-go records that another engineer can reproduce.
Before you start
- Complete Week 16's robot–edge–cloud boundaries, mission API, task allocation, operator HMI, telemetry, canary deployment, rollback, observability, and fault-injection acceptance work.
- Bring one versioned FleetOps baseline with a documented start command, two simulated robots, immutable code and configuration identities, a known-good mission, structured logs, and a reversible deployment path.
- Be able to distinguish requested, accepted, commanded, measured, and terminal mission state; keep one correlation identity across API, allocator, adapter, robot, and evidence records.
- Keep all exercises in a simulator, software emulator, unpowered design review, or approved restricted-energy HIL bench. A classroom test plan never authorizes live machinery, bypassed safeguards, or safety-function validation without qualified supervision.
By the end
- Convert requirements and hazards into measurable acceptance criteria, failure modes, controls, test cases, results, and unresolved-risk records with bidirectional traceability.
- Place pytest, gtest, ROS 2 launch_testing, integration, and complete-mission checks at the boundary where each can expose a distinct defect instead of treating test count as system confidence.
- Build a version-pinned Gazebo or Isaac scenario matrix whose fixed cases, documented seeds, assertions, artifacts, and measured reruns separate repeatable regression from robustness exploration.
- Reuse one fault contract across SIL and HIL, inject bounded sensor, clock, transport, and device failures, and report detection time, response, terminal state, and evidence limits for each test level.
- Test VDA 5050 v3 and Open-RMF adapter contracts without confusing warehouse intent, fleet traffic, vendor navigation, PLC equipment control, or independently engineered safety ownership.
- Explain the scope of ISO 3691-4 and ISO 13849, distinguish application stops from validated protective functions, and create an awareness-level safety-function table for qualified review.
- Run an evidence-linked FAT/SAT and commissioning rehearsal, preserve incident facts separately from hypotheses, prove corrective-action regression and rollback, and record an explicit FleetOps go/no-go decision.
The field story
Prove the narrow-aisle night shift
Week 16 ended with a functioning FleetOps baseline, but a working demo is not an acceptance package. A customer now asks whether Courier-A and Courier-B can survive stale localization, delayed messages, a dock sensor failure, blocked traffic, an adapter mismatch, and a release rollback during the night shift. Begin with requirement R-17: no mission may enter the shared aisle when authority or occupancy evidence is stale. Carry that single requirement through risk analysis, automated tests, simulation, SIL, HIL, interface checks, and commissioning evidence.
The mission is a proof chain, not a test-count contest. Each level must expose a distinct defect and state what it cannot represent. Deterministic scenarios use pinned worlds and replayable seeds; robustness runs remain separate. HIL exercises real interfaces against controlled emulators without pretending to reproduce payload, floor, wiring, or human hazards. Safety standards remain under qualified ownership. The final FAT/SAT record preserves facts, hypotheses, corrective actions, rollback, residual risk, and an explicit go or no-go decision that Week 25 can later operationalize.
- Why this chapter now
Week 16 supplied a versioned distributed robot service; it now needs risk-linked evidence across software, simulation, interfaces, restricted hardware boundaries, and customer acceptance.
- Ignore for now
Do not certify functional safety, energize unapproved machinery, or inflate passing test counts. Qualify the declared simulated and restricted-energy FleetOps scope.
- This unlocks
The acceptance ladder and traceability records become reusable gates for later VLA systems, capstone release, and Week 25 production operations.
- Proof you will leave with
Provide requirements and FMEA links, test-layer rationale, deterministic scenario manifests, seeds and replay, SIL/HIL fault contracts, adapter conformance cases, awareness-level safety table, FAT/SAT records, incident timeline, corrective-action regression, rollback evidence, and signed decision.
Environment contractRepository-supported Node.js 22.13.0 or newer runs the evidence-gate starter. The system baseline is the Week 16 ROS 2 Jazzy and Gazebo Harmonic FleetOps build; HIL work is limited to approved restricted-energy interfaces or emulators.
- Compatibility boundary
Gazebo or alternative simulator worlds, ROS testing packages, Open-RMF adapters, VDA 5050 interfaces, PLC/WMS boundaries, and hardware emulators change independently. Pin every tested artifact and do not infer conformance from a nearby revision.
- Smoke check
Run
node week-17-fleetops-release-gate.mjs; then execute one known-good Week 16 mission, one stale-authority rejection, one deterministic replay, and one rollback using immutable code and configuration identities.- Contract reviewed
2026-07-25
- Runtime evidence
The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
- Drift risk
high
Today in the field story
One problem, then the next
Translate the night-shift promise into measurable requirements, beginning with R-17’s stale-authority prohibition. Connect hazards, initiating conditions, failure modes, effects, controls, test cases, evidence, owners, and residual risk without collapsing them into one column. A high risk number does not prove a cause, and an implemented control does not prove effectiveness. Freeze the traceability identity that will follow the requirement through every later gate.
- Why now
Testing needs explicit claims and risks before a passing result can mean anything.
- Ignore today
Ignore exhaustive certification analysis; build a reviewable FleetOps requirement and FMEA set.
- Unlocks next
Acceptance criteria that determine which tests and evidence are necessary.
Understand
Build the physical picture first
A validation plan is a chain of custody from an observable promise or hazard to one owned control, one executable challenge, one measured result, and one explicit residual decision.
Start with statements another person can falsify. “The robot stops quickly when localization is lost” is not testable until the trigger, initial speed, detection deadline, commanded response, measured terminal condition, operating surface, payload, and evidence clock are declared. Functional requirements describe observable behavior; non-functional requirements bound qualities such as latency, availability, or freshness. Give every requirement a stable identifier so later design, test, result, deviation, and approval records do not drift apart.
A hazard is a potential source or situation of harm, while a failure mode describes how a component or function can fail. FMEA records the failure mode, local and system effects, possible causes, existing controls, detection, and follow-up action. A severity–occurrence–detection product can help sort review work, but it is not a safety argument: low estimated occurrence cannot automatically make a catastrophic consequence acceptable, and teams must not tune scores to obtain a preferred rank.
Traceability should work in both directions. Starting from a requirement or risk control, a reviewer can find its design owner, verification method, exact configuration, result, evidence, deviation, and current disposition. Starting from a test, the reviewer can name the requirement or risk it covers. An orphaned high-risk requirement is a gap; an automated test with no stated purpose may still be useful, but its pass cannot be counted as risk coverage.
Plan negative and recovery behavior before executing the happy path. Include normal load, boundary values, foreseeable misuse, stale or missing data, process death, communication loss, restart, cancellation, rollback, and post-recovery state. Freeze acceptance thresholds before the run, preserve failing evidence, and separate verification from validation: verification asks whether the implementation meets its specification, while validation asks whether the integrated system is suitable for the intended use and environment.
Words you need
Name each idea precisely
- Testable requirement
A uniquely identified statement with observable conditions, behavior, limits, units, and a decidable pass or fail result.
Physical example:REQ-LOC-04 states that a moving AMR presented with localization age above 200 ms must enter the declared stopped state within 500 ms under the named test conditions.
- Hazard
A source or situation with the potential to cause injury, damage, or another unacceptable consequence.
Physical example:Continued motion while position is unknown can expose a nearby person or fixture to collision.
- Failure mode
A specific way a component, interface, or function can cease to provide its intended behavior.
Physical example:The localization process remains alive but republishes one frozen pose timestamp, so a heartbeat-only monitor sees no failure.
- FMEA
A structured analysis that links possible failure modes to effects, causes, controls, detection, ownership, and actions.
Physical example:A row for frozen localization records collision exposure, timestamp-age monitoring, the required response, a fault-injection test, and an unresolved diagnostic gap.
- Bidirectional traceability
The ability to navigate from requirement or risk to evidence and from every claimed test result back to its purpose and governing version.
Physical example:Clicking HAZ-07 reaches control CTRL-12, tests SIL-18 and HIL-06, their artifacts, one deviation, and the signed residual-risk disposition.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalRisk priority, trial denominators, and recovery time
Validation turns hazards into traceable tests and reports every planned trial, including the failures that make a result uncomfortable.
- RPN = S×O×D
- an ordinal FMEA priority from severity, occurrence, and detection ratingsUnit: relative score
- p̂ = k/N
- observed passes k divided by all planned trials NUnit: fraction or percent
- Tᵣ
- time from a declared failure event until every recovery condition remains trueUnit: seconds (s)
A hazard is rated severity S=5, occurrence O=2, and detection difficulty D=4, so its relative RPN is 5×2×4 = 40.
A frozen matrix planned N=20 trials and passed k=17, so the observed pass rate is 17/20 = 85%; the three failures stay in the denominator.
If valid sensing returns at 12.0 s and all stability criteria hold from 15.5 s onward, report recovery time Tᵣ = 3.5 s and preserve the trace.
Treat it like a release test matrix with trace IDs, except the failed requirement can concern motion, collision, or loss of control rather than a screen defect.
A frozen suite passes 27 of 30 planned trials. What observed pass rate must be reported?
27/30 = 0.90 = 90%, with all three failures retained and categorized.
A simple priority number is
The examples give
Arithmetic ranking does not override a separate catastrophic-hazard rule.
Trace a frozen-localization risk into a decidable test
FleetOps accepts navigation only while localization age is at most 200 ms. The review team is considering a stale-pose failure with severity 9, occurrence 3, and detection difficulty 7 on internal 1–10 planning scales.
Create HAZ-07 for collision exposure during motion with unknown pose, then create FM-07A for a publisher that repeats an unchanged pose timestamp while its process remains alive.
Calculate the planning score
9 × 3 × 7 = 189, record the scale definitions, and state that 189 prioritizes review but does not decide whether the hazard is acceptable.Write REQ-LOC-04: when pose age first exceeds 200 ms during the declared simulated motion case, FleetOps rejects further navigation and reaches its specified stopped mission state within 500 ms.
Link the requirement to an independent freshness watchdog, a cancellation path, operator-visible fault state, and a qualified review item for any separate safety-rated protective function.
Define SIL-18 to freeze the source timestamp, capture the injection time and one monotonic evidence clock, and assert detection latency, command rejection, terminal mission state, process health, and recovery behavior.
Record the exact build, configuration, world, initial state, logs, trace identity, measured latency, pass or fail, reviewer, and any deviation; link every artifact back to HAZ-07 and REQ-LOC-04.
The former phrase “stop safely” becomes a reproducible software requirement and test while the qualified functional-safety decision remains explicitly outside this exercise.
Traceability makes every acceptance claim inspectable and prevents a convenient passing test from silently substituting for an untested hazard control.
Physical examples
Where this appears in real life
Fire-door promise versus inspection record
A paper sign says a model fire door closes after release. A second sheet names the release condition, maximum closing time, obstruction case, inspection date, measured result, and failed latch correction.
The first sheet states intent; the second makes the trigger, limit, adverse case, evidence, and unresolved defect inspectable without claiming the paper model is a certified door.
Bicycle brake risk ledger
With the bicycle stationary and unpowered, list worn pad, loose cable, wet rim, and stuck lever as different failure modes, then link each to its effect, detection, maintenance action, and verification.
Several mechanisms can produce weak braking, so one generic “brake tested” checkbox loses the causal detail needed to choose and verify a control.
Hands-on exercise
Make the idea observable
Use the Week 16 simulated FleetOps baseline and a local table or issue tracker. Do not move real hardware or assign performance levels, protective-field distances, or residual safety acceptance yourself.
Write ten stable requirements covering mission creation, duplicate request, cancellation, stale robot state, allocation, blocked route, operator indication, rollback, evidence retention, and recovery.
Create five failure-mode rows with effects, causes, current controls, detection gaps, planning scores where useful, owners, and required actions; include at least one severe item that cannot be accepted by arithmetic ranking.
Build a traceability table linking each high-priority requirement or control to design location, component check, integration check, complete-mission scenario, result placeholder, and deviation field.
Rewrite every ambiguous word such as fast, safe, reliable, immediate, nearby, or normal into a condition, observable quantity, unit, threshold, and time window.
Ask another engineer to start from two tests and identify their requirements, then start from two hazards and identify their evidence; repair every broken forward or backward link.
Freeze the review version, date, assumptions, exclusions, and unresolved items before any acceptance run so thresholds cannot move after results are visible.
The table exposes missing owners, requirements with no negative case, automated checks that cover no declared risk, and severe controls whose evidence is still only planned.
Every high-priority risk and requirement reaches a versioned planned test and disposition field in both directions, while ambiguous language and unsupported safety claims are removed.
Build today
Create a risk-linked SIL→HIL acceptance ladder for FleetOps, automate regression scenarios, integrate one external fleet or PLC boundary, and publish FAT/SAT evidence plus an incident report.
Evidence to save
DONE when the learning log explains “Requirements, risk, FMEA, traceability, and test planning” in five precise points and a checked example produces the predicted output.
Common mistakes
Catch the wrong mental model
Writing a test procedure before defining the requirement or risk it is meant to cover.
Give the observable promise or control a stable identity first, then link the procedure, threshold, configuration, result, deviation, and decision to that identity.
Using an RPN threshold as automatic permission to accept a severe hazard.
Use scoring only to support prioritization; apply the organization’s hazard rules, applicable standards, qualified judgment, control hierarchy, validation, and documented residual-risk process.
Removing a failed case after the implementation is corrected.
Retain the original failure evidence, corrective-action identity, new regression result, and approval so the causal history remains reviewable.
Job connection
How this becomes employable evidence
Lead a FleetOps verification review that turns warehouse workflows, communication failures, robot-state freshness, rollback, and commissioning hazards into owned requirements, controls, executable cases, evidence identities, deviations, and residual decisions.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robot Fleet Backend / Platform Engineer
Chapter 17 interview drill
Interview questions: Requirements, risk, FMEA, traceability, and test planning
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A product manager says an AMR must stop safely when localization fails. Show how you would rewrite that statement, distinguish hazard from failure mode, use FMEA without worshipping its score, and prove traceability in both directions.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What makes a requirement testable?
It names a stable condition, observable behavior, limits and units, evidence method, and a pass or fail rule another person can apply.
Q2Why can a lower FMEA score not automatically clear a catastrophic failure mode?
The product is a planning aid whose scales and estimates are imperfect; severity rules, standards, controls, validation, and qualified residual-risk acceptance remain separate decisions.
Q3What does reverse traceability reveal?
It shows whether a test result actually supports a named requirement or risk control rather than adding an impressive but irrelevant pass count.
Chapter references
- ROS 2 Jazzy — launch_testingOfficial runtime-test semantics for active tests, post-shutdown checks, process output, exit codes, time-bounded waiting, launch arguments, and ament CMake integration.
- Gazebo Sim 8 — Python TestFixtureMaintainer API for loading a versioned SDF world, stepping a Gazebo server, and checking entities and components in executable simulation fixtures.
- VDA 5050 — official version 3 specification and schemasOfficial VDA and VDMA source for the versioned mobile-robot-to-fleet-control interface, normative document precedence, JSON schemas, messages, actions, and compatibility review.
- Open-RMF — Fleet Adapter TutorialOpen-RMF maintainer guidance for the adapter bridge, robot state and battery updates, task dispatch, vendor API callbacks, coordinate transforms, navigation graphs, and simulation time.
- OSHA Technical Manual — Industrial Robot Systems and SafetyGovernment guidance on documented risk assessment before commissioning, site acceptance against utilities and interfaces, safety-function verification, training, safeguards, and periodic revalidation.