Chapter 17 · Prove FleetOps across regression, interfaces, safety, and commissioning
Today in the field story
One problem, then the next
Freeze the aisle world, robot starts, mission manifest, fault schedule, assertions, artifacts, and selected seeds. Re-run the stale-localization and blocked-aisle cases to establish deterministic regression, then separate additional randomized robustness exploration. A seed is evidence only with the exact simulator, assets, physics, and code identity. Preserve replay commands so another engineer can reproduce the same disputed transition.
- Why now
System requirements need repeatable environments before failures can be compared across changes.
- Ignore today
Ignore claiming simulator determinism across arbitrary platforms; pin and measure this matrix.
- Unlocks next
Replayable system cases suitable for SIL and later failure comparison.
Understand
Build the physical picture first
A simulation regression case is a frozen experiment envelope: world, robot, software, parameters, start state, seed, fault, oracle, and artifact schema travel together.
A simulator run is reproducible only to the level its complete environment permits. Pin the Gazebo SDF or Isaac USD assets, simulator and physics versions, robot model, plugins, ROS middleware and bridge, parameters, step size, sensor configuration, initial state, and test code. Gazebo’s TestFixture can load a world, step the server, and inspect entities and components; an Isaac workflow can likewise run a versioned standalone scene with explicit randomization seeds. Treat both as executable environments, not videos.
A fixed seed controls only the random generators that actually use it. Thread scheduling, GPU kernels, physics implementation, middleware arrival order, asynchronous writers, external services, and version changes can still alter timing or results. Run important fixed cases repeatedly and measure result stability. Call a test deterministic only for the declared observable and tolerance; use “repeatable under this manifest” when bit-for-bit identity has not been demonstrated.
Separate regression from robustness. A regression set holds scenario definitions and expected outcomes stable so a software change can be compared with a known baseline. A robustness set deliberately varies pose, friction, payload, latency, lighting, obstacles, or seeds within justified ranges. Randomizing everything at once makes failures difficult to localize, while testing one perfect world creates unjustified confidence about deployment diversity.
Replay is more than rerunning a bag or watching a recording. Preserve the scenario identity, commands, input streams, world and model versions, clock behavior, fault schedule, expected oracle, software bundle, logs, traces, and terminal measurements. A bag can replay ROS inputs but cannot recreate changed physics, contact, plugin state, external database behavior, or nondeterministic scheduling unless those dependencies are also captured or replaced by an explicitly scoped fixture.
Words you need
Name each idea precisely
- Scenario manifest
The versioned description of every environment, model, start, parameter, seed, fault, oracle, and artifact needed to identify one test case.
Physical example:SCN-BLOCKED-03 pins the warehouse world digest, robot model, two start poses, obstacle placement, seed 41, blocked-lane event time, and recovery deadline.
- Scenario seed
A value used to initialize participating pseudo-random generators so their generated sequence can be repeated under compatible execution.
Physical example:Seed 41 recreates one obstacle placement sequence but does not force GPU scheduling or DDS delivery order to become identical.
- Regression matrix
A stable set of named scenarios used to detect behavior changes between frozen software and configuration baselines.
Physical example:Every release reruns nominal delivery, blocked aisle, low battery, stale state, cancellation, and adapter restart with unchanged acceptance rules.
- Robustness sweep
A declared variation of plausible conditions used to estimate how behavior changes across deployment diversity.
Physical example:The same docking task runs across three payloads, two floor-friction estimates, and five start offsets selected before results are seen.
- Replay bundle
The inputs and versioned environment context required to reproduce or closely reconstruct a recorded failure and evaluate the same oracle.
Physical example:A bundle contains MCAP, manifest, world digest, parameters, container digest, fault timeline, command, expected terminal state, and trace links.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalRisk priority, trial denominators, and recovery time
Validation turns hazards into traceable tests and reports every planned trial, including the failures that make a result uncomfortable.
- RPN = S×O×D
- an ordinal FMEA priority from severity, occurrence, and detection ratingsUnit: relative score
- p̂ = k/N
- observed passes k divided by all planned trials NUnit: fraction or percent
- Tᵣ
- time from a declared failure event until every recovery condition remains trueUnit: seconds (s)
A hazard is rated severity S=5, occurrence O=2, and detection difficulty D=4, so its relative RPN is 5×2×4 = 40.
A frozen matrix planned N=20 trials and passed k=17, so the observed pass rate is 17/20 = 85%; the three failures stay in the denominator.
If valid sensing returns at 12.0 s and all stability criteria hold from 15.5 s onward, report recovery time Tᵣ = 3.5 s and preserve the trace.
Treat it like a release test matrix with trace IDs, except the failed requirement can concern motion, collision, or loss of control rather than a screen defect.
A frozen suite passes 27 of 30 planned trials. What observed pass rate must be reported?
27/30 = 0.90 = 90%, with all three failures retained and categorized.
The full matrix contains
At each, serial time is
Size a frozen simulation matrix and its smoke subset
The planned full suite has 4 worlds, 3 robot starts, 2 documented seeds, and 3 fault modes. One case takes 50 seconds after startup, and the CI smoke budget is 8 minutes.
Calculate the full matrix as
4 × 3 × 2 × 3 = 72named cases; do not describe one ad hoc loop as 72 independent requirements.Estimate serial scenario time as
72 × 50 s = 3,600 s = 60 min, then add measured startup, teardown, artifact, and retry overhead separately.Choose a predeclared smoke set of 8 cases covering every fault at least once, the most critical world, boundary starts, and both seeds;
8 × 50 s = 400 sleaves 80 seconds for overhead.Run each smoke case twice under one pinned manifest and compare terminal state, outcome, and tolerance-bounded timing rather than demanding unsupported byte identity.
Reserve the 72-case full matrix for the gated lane with controlled parallel workers, isolated namespaces, resource limits, and deterministic result ordering.
If one repeated smoke case changes outcome, mark the gate failed or flaky, preserve both bundles, and investigate scheduling, physics, seed coverage, external state, and oracle instability before expansion.
The eight-case smoke lane fits the nominal eight-minute budget, while the complete 72-case, one-hour serial estimate remains visible and scheduled for the gate.
A useful matrix balances risk coverage, runtime, isolation, and repeatability without shrinking the full acceptance obligation into the fastest subset.
Physical examples
Where this appears in real life
Marked tabletop obstacle course
Tape fixed start, goal, and obstacle outlines on a table, then move an unpowered cardboard robot token through the course using a written move list before changing exactly one obstacle.
The baseline is comparable only while layout, start, commands, and scoring stay fixed; the changed obstacle belongs to a new named condition rather than a silent rerun.
Recipe and oven variation
Two cards record ingredients, quantities, oven setting, tray position, timer, and result, while a third experiment deliberately changes only tray position.
A recipe resembles the manifest, repeated batches measure repeatability, and a controlled variation tests sensitivity without hiding which factor changed.
Hands-on exercise
Make the idea observable
Use either the existing Gazebo FleetOps world or an available Isaac Sim equivalent. If only one simulator is installed, execute there and document the other as unverified rather than inventing cross-simulator parity.
Create a manifest that pins simulator, physics and bridge versions, world or USD digest, robot model, containers, ROS distribution, middleware, parameters, time step, sensor settings, initial poses, and seed ownership.
Define six fixed scenario IDs: nominal delivery, duplicate request, blocked route, low battery, stale robot state, and cancellation during motion; give each an independent terminal oracle.
Run every fixed case twice, compare outcomes and bounded timing, and record process order, CPU or GPU context, and any observable nondeterminism.
Create a separate robustness sweep that changes one plausible factor at a time—start offset, payload estimate, latency, friction, or obstacle placement—and preserve the chosen range rationale.
Plant one wrong initial pose and one stale world digest; make the harness reject both before execution instead of producing incomparable results.
Package command, manifest, logs, traces, MCAP where useful, fault timeline, terminal measurements, pass or fail, and known replay limits under one immutable result identity.
Pinned cases reveal software regressions, repeated runs reveal instability, controlled variations expose sensitivity, and manifest guards prevent silent environment changes from masquerading as product behavior.
Another engineer runs one command, reproduces all fixed scenario identities and both manifest rejections, and can explain every observed difference without relying on a screen recording.
Build today
Create a risk-linked SIL→HIL acceptance ladder for FleetOps, automate regression scenarios, integrate one external fleet or PLC boundary, and publish FAT/SAT evidence plus an incident report.
Evidence to save
DONE when a deterministic “Deterministic Gazebo or Isaac scenario matrices, seeds, and replay” failure test reports expected versus actual behavior and passes after the documented fix.
Common mistakes
Catch the wrong mental model
Claiming a fixed seed guarantees identical simulation.
Identify every seeded generator, pin the environment, repeat the case, compare declared observables with tolerances, and report remaining scheduling, GPU, physics, middleware, or external-state nondeterminism.
Using randomized tests as the only regression gate.
Keep named fixed cases for change detection and run predeclared bounded variations separately for robustness, with every realized parameter recorded.
Saving only an MCAP file or demo video after a failure.
Preserve the versioned world, models, start state, seed and fault schedule, configuration, software digests, inputs, oracle, logs, traces, and terminal measurement needed to reconstruct the experiment.
Job connection
How this becomes employable evidence
Own a version-pinned Gazebo or Isaac regression service that schedules isolated fleet scenarios, measures repeated outcomes, captures failure bundles, distinguishes robustness sweeps from frozen gates, and rejects environment drift before test execution.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robotics Application / ROS 2 Integration Engineer
Chapter 17 interview drill
Interview questions: Deterministic Gazebo or Isaac scenario matrices, seeds, and replay
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A simulation test passes with seed 42 on one machine and fails in CI. Explain what a seed controls, what it does not, which versions and clocks you would inspect, and how you would preserve a replayable failure bundle.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What does a simulator seed fail to control by itself?
Unseeded generators, thread and process scheduling, GPU behavior, physics or middleware version changes, asynchronous I/O, external services, and different operation order can still vary.
Q2Why keep fixed regression cases separate from a robustness sweep?
Fixed cases make release-to-release changes comparable; controlled variation measures sensitivity across plausible conditions without obscuring which factor changed.
Q3What must accompany replayed ROS messages to recreate a physics failure?
The matching world, robot and plugin versions, parameters, initial state, clocks, fault schedule, software bundle, external dependencies, and the original outcome oracle.