Chapter 17 · Prove FleetOps across regression, interfaces, safety, and commissioning
Today in the field story
One problem, then the next
Reuse one fault contract across SIL and an approved HIL emulator: injected sensor value, start time, duration, expected detection, bounded response, terminal state, and evidence. Challenge clock drift, packet loss, stuck dock input, and device restart. Report what the emulator proves about real interfaces and timing, then state the payload, wiring, floor, contact, and human hazards it cannot reproduce.
- Why now
The validation ladder must increase realism without erasing each level’s evidence limits.
- Ignore today
Ignore unrestricted hardware testing; compare controlled SIL and HIL responses.
- Unlocks next
A shared fault vocabulary and measured promotion gate between test levels.
Understand
Build the physical picture first
SIL and HIL are adjacent rungs on one evidence ladder: the scenario contract stays stable while the real interfaces, timing, power, and plant behavior introduced at each rung are named.
Software-in-the-loop runs production or near-production application logic against simulated robots, sensors, actuators, clocks, and environments. It is fast enough for broad regression and destructive fault injection, but its timing, device, contact, load, power, and mechanical evidence are model-bounded. Hardware-in-the-loop introduces real controllers, compute, buses, I/O, or device interfaces while the plant or some sensors remain controlled by an emulator. State exactly which hardware is real; a laptop connected to a simulated robot is not automatically HIL.
A sensor emulator needs a contract as rigorous as a driver: message or electrical interface, units, range, frame, rate, timestamp source, noise model, startup, reset, disconnect, saturation, corruption, and stale-data behavior. Plausible-looking values can be more dangerous than obvious absence. Freeze a timestamp while continuing to publish, repeat a sequence number, hold one channel constant, jump time backward, and send a validly encoded out-of-range value so consumers must check semantics instead of heartbeat alone.
Inject faults at a known boundary and record one evidence clock. Useful cases include latency, packet loss, duplication, reordering, checksum error, delayed acknowledgment, clock jump, process restart, bus disconnect, and device reset. Define the expected detection, bounded response, terminal state, operator indication, and recovery rule before injection. Never inject on live production equipment or bypass protective systems; use an approved bench, restricted energy, isolation, and a qualified test owner.
HIL closes some gaps but not all. It can expose real CPU load, scheduler behavior, firmware, bus timing, connector faults, driver semantics, and power transients present in the bench. An emulated motor does not prove braking distance, payload behavior, cable flex, tire friction, actuator heating, collision response, or protective fields. Keep a coverage ledger that states what SIL proves, what HIL adds, and which restricted-energy real-system or site tests remain blocked.
Words you need
Name each idea precisely
- Software-in-the-loop
Execution of application software against simulated plant, sensors, devices, or environment through declared production-like interfaces.
Physical example:The real FleetOps allocator and ROS adapter command two Gazebo robots while sensor and motion results come from simulation.
- Hardware-in-the-loop
A controlled test that includes named real controllers, compute, I/O, buses, or device interfaces while another part of the physical plant remains simulated or emulated.
Physical example:The production edge computer and CAN interface receive encoder frames from a fault-capable bench emulator while motor power remains isolated.
- Fault injection point
The exact interface and direction where a test deliberately introduces one declared abnormal condition.
Physical example:A proxy between sensor emulator and ROS driver freezes source timestamps while preserving the nominal 40 Hz arrival cadence.
- Data-age limit
The maximum permitted age of data relative to the clock and physical event used by the consumer.
Physical example:A controller rejects lidar state whose acquisition timestamp is more than 125 ms behind its monotonic decision clock.
- Coverage ledger
A record of the real and emulated elements, challenged faults, measured results, and untested physical effects at each evidence level.
Physical example:The HIL row confirms real edge-compute timing and bus reset recovery but marks braking, payload, floor friction, and scanner fields unverified.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalRisk priority, trial denominators, and recovery time
Validation turns hazards into traceable tests and reports every planned trial, including the failures that make a result uncomfortable.
- RPN = S×O×D
- an ordinal FMEA priority from severity, occurrence, and detection ratingsUnit: relative score
- p̂ = k/N
- observed passes k divided by all planned trials NUnit: fraction or percent
- Tᵣ
- time from a declared failure event until every recovery condition remains trueUnit: seconds (s)
A hazard is rated severity S=5, occurrence O=2, and detection difficulty D=4, so its relative RPN is 5×2×4 = 40.
A frozen matrix planned N=20 trials and passed k=17, so the observed pass rate is 17/20 = 85%; the three failures stay in the denominator.
If valid sensing returns at 12.0 s and all stability criteria hold from 15.5 s onward, report recovery time Tᵣ = 3.5 s and preserve the trace.
Treat it like a release test matrix with trace IDs, except the failed requirement can concern motion, collision, or loss of control rather than a screen defect.
A frozen suite passes 27 of 30 planned trials. What observed pass rate must be reported?
27/30 = 0.90 = 90%, with all three failures retained and categorized.
The nominal sensor period is
A age limit spans
nominal periods; the test pause is .
Derive and test a stale-sensor deadline
A sensor contract specifies 40 Hz nominal publication and a 125 ms maximum acquisition age. The fault harness can continue publishing while freezing the source timestamp for 150 ms.
Convert rate to nominal period:
1 / 40 Hz = 0.025 s = 25 ms.Express the freshness budget in periods:
125 ms / 25 ms = 5nominal periods, noting that age is decided from timestamps rather than packet count.At injection time
t0, freeze the acquisition timestamp but continue publishing well-formed messages at 40 Hz so a heartbeat-only monitor stays green.Assert that once measured age exceeds 125 ms, the consumer rejects the sample, reports the named stale-data fault, stops issuing new motion requests, and reaches the declared fallback within its own deadline.
Because the freeze lasts 150 ms, verify at least one decision occurs beyond the age limit; record detection latency from the monotonic harness clock and preserve all message timestamps.
Repeat the same scenario in SIL and, if an approved bench exists, HIL; compare detection and response timing while listing physical plant effects neither level exercises.
The injected stream looks alive yet crosses the five-period freshness limit, so a semantic watchdog must expose the fault in both evidence levels.
A strong emulator reproduces convincing wrongness, not only total disconnection, and the oracle measures the consumer’s safe response from a known fault boundary.
Physical examples
Where this appears in real life
Controller bench without motor power
A labelled diagram places real edge compute and a real bus interface beside an emulator that supplies encoder frames, delayed acknowledgments, and disconnect events while the motor-power path stays isolated.
The test can expose driver, firmware, bus, and timing behavior, but the diagram prevents anyone from inferring real torque, braking, thermal, or contact evidence.
Stopped clock with arriving envelopes
A person delivers a new envelope every second, but every enclosed card repeats the same old timestamp and position value.
Arrival heartbeat remains healthy while information freshness fails, illustrating why consumers check source time and semantic progress rather than packet presence alone.
Hands-on exercise
Make the idea observable
Begin in SIL. Use HIL only with an approved named bench, isolated actuator energy, documented wiring, emergency procedure, and responsible supervisor; otherwise produce the HIL-ready scenario and mark execution blocked.
Define one scenario schema containing level, real elements, emulated elements, injection point, precondition, fault parameters, evidence clock, expected detection, response deadline, terminal state, reset, and artifact paths.
Implement or configure seven bounded cases: frozen timestamp, backward clock jump, noise burst, packet loss, delayed acknowledgment, device reset, and validly encoded out-of-range value.
Run the cases in SIL, preserving per-case injection time, observed detection, first response, terminal state, operator indication, recovery, and exact software and simulator identities.
Map the same schema to the available HIL interfaces without weakening its oracle; if one fault cannot be generated safely or faithfully, record the reason and substitute no hidden software shortcut.
Compare SIL and HIL timing distributions and failure semantics, then list newly covered real behaviors and remaining load, contact, braking, thermal, wiring, and environmental gaps.
Verify every fault leaves the bench or simulator in a known reset state before the next case and that cancellation cannot accidentally re-enable motion or reuse stale state.
The shared schema exposes where the two levels behave differently, while the coverage ledger prevents real controller timing from being mistaken for proof of the full mechanical system.
All seven SIL faults have decisive artifacts, every available HIL case preserves the same oracle, blocked HIL cases name their exact missing capability, and no result overclaims real-robot acceptance.
Build today
Create a risk-linked SIL→HIL acceptance ladder for FleetOps, automate regression scenarios, integrate one external fleet or PLC boundary, and publish FAT/SAT evidence plus an incident report.
Evidence to save
DONE when the integrated “SIL, HIL, sensor emulators, time faults, and communication faults” path is observable, cancelable, and leaves the prior baseline reproducible.
Common mistakes
Catch the wrong mental model
Calling any test involving a physical computer HIL.
Name the real controller, I/O, bus, firmware, or device interface and the emulated plant; otherwise classify the level by the boundary actually exercised.
Injecting only complete sensor disconnection.
Also challenge plausible stale, duplicated, reordered, delayed, corrupt, saturated, and out-of-range data so semantic validation and fallback behavior are exercised.
Treating a HIL pass as proof of braking or collision safety.
Limit the claim to the named bench hardware and interfaces, then preserve real load, friction, brake, protective-field, contact, thermal, and environmental tests as separate qualified obligations.
Job connection
How this becomes employable evidence
Build a reusable fault harness that drives the same sensor, clock, transport, and reset scenarios through simulation and an isolated controller bench, then publishes latency comparisons and an honest real-versus-emulated coverage ledger.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 17 interview drill
Interview questions: SIL, HIL, sensor emulators, time faults, and communication faults
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Define SIL and HIL for a mobile robot without hand-waving. Show how you would freeze a timestamp while packets continue, measure the response, and explain which real-world failures still require restricted-energy or site validation.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What must be named before a test can credibly be called HIL?
The real hardware and interfaces, emulated plant or devices, fault boundary, energy isolation, expected behavior, measurements, and remaining physical gaps.
Q2Why continue publishing while freezing a source timestamp?
It proves whether the consumer detects semantic staleness instead of relying only on message arrival or process heartbeat.
Q3Which evidence gap remains after a controller-board HIL test?
Actual actuator, brake, payload, friction, contact, cable, thermal, protective-field, and site behavior remain unproven unless the bench genuinely includes and measures them.