Chapter 07 · Build a repeatable robot laboratory in Gazebo
Today in the field story
One problem, then the next
Build the Rainy-Day Simulator Lab’s sensor monitor as a state machine that checks presence, timestamp progression, age, rate, plausibility, transform availability, and cross-sensor consistency. Inject frozen timestamps and valid-looking stale values, then verify the simulated command response and guarded recovery. A red dashboard without a changed command path is observation, not degraded behavior.
- Why now
The lab must convert measured evidence quality into an explicit bounded operational state.
- Ignore today
Ignore calling software health logic a certified stop system.
- Unlocks next
Fault verdicts that the final scenario matrix can score objectively.
Understand
Build the physical picture first
Sensor health is a chain of questions—present, fresh, plausible, geometrically usable, and consistent—followed by a planned response when any answer becomes no.
A topic name in the ROS graph means only that an endpoint was discovered. It does not prove new measurements are arriving, timestamps advance, values are finite, frames connect, or the scene is believable. Build health from separate checks: expected publisher and type, receipt rate, stamp progression, age, numeric range, invalid fraction, sequence continuity, transform availability, and agreement with other evidence. Keep the raw facts visible so an operator can distinguish no publisher, stale data, corrupt data, and a blocked sensor.
Health limits come from task risk and sensor contract. A 200 ms-old camera might be acceptable for a parked inspection image but dangerous for teleoperation near a person. A lidar dropout may require a mobile robot to stop, while a temporarily missing rear camera might permit slow forward-only retreat if an independently reviewed safety design supports it. Simulation lets you test the software response, but it cannot certify that the response satisfies machinery safety rules. Name the allowed behavior, speed, direction, timeout, and recovery condition explicitly.
Use a small state machine rather than one flickering boolean. A sensor can be INITIALIZING, HEALTHY, DEGRADED, FAILED, or RECOVERING. Hysteresis prevents a value near one threshold from rapidly switching states: enter degraded after several bad periods, and require several good periods before recovery. Latching a serious fault until operator acknowledgment may be appropriate. Never silently substitute zero for an invalid measurement because zero could mean an obstacle touching the robot or no motion, depending on the quantity.
A useful operator and log view shows last measurement stamp, age, observed rate, invalid count, frame error, health state, reason, response, and last transition time. The command path should consume the health state, not merely colour an HMI after commands continue unchanged. Validate each fallback by injecting one known fault and checking detection time, bounded command, terminal state, recovery gate, and false-positive behavior. A degraded mode that has never been exercised is only a design claim.
Words you need
Name each idea precisely
- Freshness
Whether an observation's age is within the limit declared for its current use.
Physical example:A scan stamped 80 ms ago passes a 100 ms navigation limit but fails a 50 ms high-speed limit.
- Plausibility check
A test that asks whether a value or change is physically and contractually possible.
Physical example:A room-temperature sensor jumping from 22 °C to 500 °C in one sample is rejected even though the message arrived on time.
- Degraded mode
A deliberately restricted operating mode used when required capability or confidence is reduced.
Physical example:A rover limits speed and refuses new autonomous goals while one non-safety camera restarts.
- Hysteresis
Different evidence requirements for entering and leaving a health state, used to prevent rapid toggling.
Physical example:Enter degraded after three stale samples but require ten consecutive fresh samples before returning healthy.
- Terminal state
The final declared result of a scenario, such as safely stopped, completed, cancelled, aborted, or timed out.
Physical example:After a frozen lidar stamp, the test expects zero measured velocity and mission result ABORTED rather than an endless active goal.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalSimulation time, real-time factor, and sensor noise
A simulator is useful only when time and uncertainty are measured honestly.
- RTF
- simulated elapsed time divided by real elapsed timeUnit: unitless
- μ
- average sensor valueUnit: sensor unit
- σ
- typical spread around the averageUnit: sensor unit
A run simulates 20 s but takes 25 s of wall time.
RTF = 20/25 = 0.8.
The simulator is slower than real time; compare timestamps in simulated time and report RTF with every performance result.
A test clock can be mocked in web software; robot sensor timestamps must still stay mutually consistent.
Ten simulated seconds take five real seconds. What is RTF?
10/5 = 2.0, so simulation runs twice as fast as real time.
A message age of exceeds the freshness limit. At , the possible travel during that stale interval is
This quantifies exposure; it does not choose the safe fallback.
Classify a frozen 10 Hz lidar
A lidar nominally publishes every 100 ms. Messages still arrive, but the measurement stamp has not advanced for 250 ms. The navigation freshness limit is 200 ms.
Calculate nominal period: 1 ÷ 10 Hz = 0.10 s = 100 ms.
Compare stamp age with the limit: 250 ms is greater than 200 ms, so the observation is stale.
Express the freeze in periods: 250 ÷ 100 = 2.5 nominal periods without a new observation time.
Keep receipt-rate status separate because callbacks may still arrive at 10 Hz with copied old stamps.
Transition according to policy, bound or stop commands, and record detection time, reason, requested command, delivered command, and measured velocity.
Require the declared number of fresh, plausible samples and transform coverage before recovery.
The lidar fails the freshness check despite normal callback rate, and the scenario should reach its defined bounded terminal state.
A heartbeat can be alive while the physical observation is dead; health needs timestamp and value semantics.
Physical examples
Where this appears in real life
Automatic door safety beam
A door controller receives no fresh clear-path evidence from its closing safety sensor.
Absence of evidence must not be treated as an empty doorway; the permitted response is determined by the safety design.
Car tyre-pressure display
One tyre reading disappears while the other three remain plausible. The dashboard reports which sensor is unavailable instead of inventing normal pressure.
Separate sensor loss, low pressure, stale last value, and the vehicle behavior allowed under each state.
Hands-on exercise
Make the idea observable
Create a small health monitor around one simulated lidar or range topic and connect its state to a bounded simulated command gate.
Define expected type, rate range, freshness limit, numeric limits, invalid fraction, transform requirement, health states, response, and recovery rule.
Publish or bridge a healthy baseline and log measurement stamp, receipt time, age, rate, value status, frame status, state, and permitted command.
Inject no publisher, frozen stamp with live callbacks, out-of-range values, and missing transform as separate scenarios.
Assert detection deadline, state transition, bounded output, measured stop or restriction, terminal result, and recovery behavior for each fault.
Build one compact operator panel or report that states the exact reason and age instead of showing only a red or green icon.
Each fault can share the same red outcome while requiring different diagnosis and recovery; the reason code and raw evidence prevent guesswork.
Every injected condition produces the expected state and physical or simulated command response, and no recovery occurs before the written evidence gate is satisfied.
Build today
Simulate a ros2_control differential-drive robot in Gazebo Harmonic with lidar, RGB-D camera, IMU, noise, bridges, and MCAP replay.
Evidence to save
DONE when a 60–120 second uncut “Health checks and degraded-sensor behavior” demo links to its command, logs or plots, result count, and honest failure note.
Common mistakes
Catch the wrong mental model
Defining health as topic present or callback received.
Check stamp progression, age, rate, validity, plausibility, frames, and task-specific consistency as separate signals.
Displaying a fault while leaving the command path unchanged.
Connect a reviewed health state to an explicit bounded response and verify the resulting measured behavior.
Returning healthy after one good sample.
Use a written recovery gate with hysteresis, transform coverage, and any required operator or subsystem acknowledgment.
Job connection
How this becomes employable evidence
Build a robot-health panel and command gate that distinguish disconnected, stale, implausible, and untransformable sensor data and prove each degraded response with fault injection.
Relevant target roles
- Robot HMI / Control & Monitoring Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
Chapter 07 interview drill
Interview questions: Health checks and degraded-sensor behavior
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A lidar indicator is green because callbacks arrive at 10 Hz, yet the robot reacts to an old obstacle layout. Design the health signals, operator display, safe response, and recovery test.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why is a normal callback rate insufficient evidence of freshness?
A publisher can repeatedly send copied data with an unchanged old timestamp, so callbacks arrive while observation age grows.
Q2What should a useful health panel show besides a colour?
The state reason, measurement stamp, age, observed rate, value or frame fault, response, and last transition time.
Q3What does hysteresis improve in a health state machine?
It prevents rapid state flicker and requires stronger sustained evidence before recovery than one boundary-crossing sample.