Chapter 05 · Build and debug a real ROS 2 system

Today in the field story

One problem, then the next

Finish the Aisle Seven Watchdog by defining heartbeat source, period, clock, identity, timeout, state machine, simulated response, and guarded recovery. Inject process loss, frozen source time, callback starvation, QoS mismatch, restart, and unsafe automatic resume. The final architecture note must show why graph presence, reliable delivery, and a ROS application watchdog are each weaker than independent physical safety.

Why now: The chapter needs one integrated failure path that connects interfaces, scheduling, lifecycle, evidence, and bounded recovery.
Ignore today: Ignore calling this an emergency stop or safety-rated controller.
Unlocks next: A production-shaped health boundary for the simulated robot body.

Understand

Build the physical picture first

A watchdog is a hall monitor with a clock, not a fortune teller. It does not assume a robot component is healthy because its name exists. It checks when trustworthy evidence last arrived, moves through explicit health states, and requests only a predefined safe simulated response when evidence becomes too old.

The week's capstone graph has at least a telemetry node, command/action node, watchdog, and observable status consumer. The watchdog receives timestamped heartbeats or meaningful state updates and calculates freshness for each monitored component. ROS discovery can show that a publisher endpoint exists, but existence is not health: a process can remain alive while publishing frozen, delayed, invalid, or semantically wrong data. Application-level monitoring must check the evidence that matters to the task.

A heartbeat contract states source, rate, timestamp clock, sequence or identity, timeout, tolerated jitter/loss, and meaning. At 10 Hz, the nominal period is 100 ms. A timeout set to exactly 100 ms will false-trigger under ordinary scheduling and network variation; select a limit from measured worst-case delay plus task risk, then test it. Use a monotonic clock for local duration measurement and handle simulated/ROS time deliberately. Wall-clock changes must not make stale data appear fresh.

Use explicit states such as STARTING, HEALTHY, DEGRADED, STALE, and RECOVERING. One late sample might enter DEGRADED; exceeding the declared age enters STALE. Recovery should not jump directly to HEALTHY after one accidental packet. Require a defined number or duration of consecutive valid samples, confirm identity/configuration, and reset task state as needed. Hysteresis prevents noisy oscillation at the threshold while still respecting the maximum unsafe age.

The watchdog's response must be designed with the controller and hazard analysis. In this chapter it can set a simulated command to zero or mark commands inhibited. On physical equipment, a ROS node must not be described as an emergency stop or safety-rated controller unless the architecture and certification actually support that claim. The watchdog can request a safe operational state through a trusted path, while independent hardware/controller safety layers bound motion if the application or network fails.

Observability turns the demo into job evidence. Every state change should log component ID, previous/new state, source timestamp, observed age, threshold, reason, and action taken. The status UI should distinguish no data, stale data, invalid data, and disconnected discovery. Test normal cadence, delay, drop, frozen timestamp, wrong identity, process kill, restart, and recovery. Record latency from injected fault to detection and simulated safe output, plus false positives during a healthy run.

Words you need

Name each idea precisely

Heartbeat: A periodic message asserting component identity and current progress/health evidence under a declared contract.
Physical example:
A motor-controller bridge publishes its mode, sequence, and source time ten times per second.
Freshness: How recent the underlying evidence is, normally calculated from now minus its trustworthy source timestamp.
Physical example:
A status stamped 150 ms ago is stale when its maximum accepted age is 120 ms.
Watchdog: An independent monitor that detects missing or stale evidence and triggers a predefined bounded response.
Physical example:
A simulated command gate inhibits new motion after telemetry exceeds its age limit.
Hysteresis: Different entry and recovery rules that stop a state from rapidly switching near one threshold.
Physical example:
Enter stale after one 150 ms age, but require three fresh samples before returning healthy.
Failure injection: A deliberate controlled test that introduces delay, loss, invalid data, or process failure to verify detection and recovery.
Physical example:
Pause heartbeat publication for 300 ms in simulation and measure the watchdog transition.
Safe operational state: A predefined application response appropriate to the system hazard and enforced by trusted controls.
Physical example:
The simulator inhibits motion commands and reports operator intervention required.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalMessage rates, queues, and latency

ROS nodes form a distributed timing system; rates and queue depth decide freshness.

λ: messages arriving each secondUnit: messages/s
μ: messages processed each secondUnit: messages/s
latency: receive time minus source timestampUnit: milliseconds (ms)

A camera publishes λ = 30 messages/s while a node processes μ = 20 messages/s.
The backlog grows by λ − μ = 10 messages each second.
A depth-5 queue fills in about 0.5 s; choose a QoS policy based on whether freshness or completeness matters.

Programmer analogy

ROS pub/sub resembles backend messaging, but an old robot message can command the wrong physical state.

Input is 50 Hz and processing is 40 Hz. How fast does backlog grow?

10 messages per second.

Heartbeat age is

t_{\mathrm{now}}-t_{\mathrm{last}}=0.35-0.20=0.15\,\mathrm{s}=150\,\mathrm{ms}.

At $f=10\,\mathrm{Hz}$ , $T=100\,\mathrm{ms}$ ; therefore a $120\,\mathrm{ms}$ timeout has been exceeded. The period sizes expectations, while explicit state logic defines the response.

Classify a stale heartbeat and verify recovery

A telemetry node publishes at 10 Hz. The watchdog declares DEGRADED above 110 ms and STALE above 120 ms. At now=0.350 s, the last trustworthy source stamp is 0.200 s. Recovery requires three consecutive samples no older than 100 ms.

Calculate nominal period: T = 1/10 = 0.100 s = 100 ms.
Calculate heartbeat age: 0.350 - 0.200 = 0.150 s = 150 ms.
Compare 150 ms with the 120 ms stale threshold and transition from HEALTHY or DEGRADED to STALE.
Record the exact reason and request the predefined simulated command-inhibit state; timestamp both detection and inhibit acknowledgement.
When publication resumes, do not recover after the first packet. Count fresh valid samples with age ≤100 ms.
After the third consecutive fresh sample and identity/configuration checks, transition STALE→RECOVERING→HEALTHY according to the declared rule.

Result

The 150 ms heartbeat is stale by 30 ms. The simulated command path is inhibited, and health returns only after three consecutive valid fresh samples.

What this proves

A watchdog needs measured time, explicit states, a bounded response, and guarded recovery; 'node present' and 'last value looks normal' are insufficient.

Physical examples

Where this appears in real life

Freezer temperature alarm

A display keeps showing -18 °C after its sensor cable is removed because the last value remains on screen.

Look for:

The number looks safe but its timestamp grows old. A freshness monitor must label it stale rather than trusting the value.

Warehouse robot command gate

A fleet service disconnects while an edge controller still has the last mission command.

Look for:

Command ownership and expiry determine whether work may continue. A watchdog inhibits new simulated commands and exposes the reason; it does not pretend that network loss is a hardware e-stop.

Hands-on exercise

Make the idea observable

Integrate the week's Python ROS 2 nodes in simulation: telemetry, SimMove command/action, watchdog, and a small console or web status consumer. All commands affect numbers only.

Launch all nodes with versioned parameters for heartbeat rate, degraded/stale thresholds, recovery sample count, namespace, and simulated command limit.
Log source and receipt timestamps, sequence, identity, current state, state-change reason, simulated command, and action goal state.
Run a five-minute healthy baseline and record message rate, maximum age, callback delay, false watchdog transitions, and CPU/process facts.
Inject one fault at a time: delayed heartbeat, dropped sequence, frozen source timestamp, wrong robot ID, killed telemetry process, and restarted process.
For every fault, measure injection→detection, detection→command-inhibit, terminal action behavior, and recovery; save the relevant graph and log slice.
Create an architecture note showing node/process boundaries, topic/service/action contracts, QoS, callback groups, lifecycle/readiness, thresholds, and which independent safety protections would be required before hardware.
Run the complete acceptance sequence from a clean launch and capture an uncut result table with PASS/FAIL and the largest remaining limitation.

Observe

Discovery loss, stale source time, invalid identity, and callback delay can all produce different evidence even when the operator symptom is 'no update'. Recovery can be more dangerous than detection if old goals resume automatically.

Done when

The repository starts from one documented command; the graph matches the architecture; every injected fault has a measured detection and simulated safe response; recovery requires fresh evidence; and the report clearly says this is not a safety-rated hardware controller.

Build today

Build a Python/C++ telemetry, command, and watchdog system; then reproduce QoS, discovery, executor, lifecycle, and cancellation failures.

Evidence to save

DONE when the weekly ship note explains how “Multi-node watchdog demo and architecture note” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Using node discovery presence as the only health signal.

Better

Check trustworthy source timestamps, sequence/progress, semantic validity, expected state, and task-relevant outputs.

Wrong

Setting timeout equal to the nominal heartbeat period.

Better

Measure scheduling/network jitter, choose a risk-based margin and degradation policy, and test false positives.

Wrong

Returning to healthy after the first packet following a failure.

Better

Use guarded recovery with consecutive fresh valid evidence, identity/config checks, and explicit task reset rules.

Wrong

Calling a ROS watchdog an emergency stop.

Better

Describe its actual application-level command gate and retain independent controller/hardware safety appropriate to the hazard.

Wrong

Testing only a killed process.

Better

Also inject delay, stale/frozen timestamps, wrong identity, callback starvation, QoS/discovery mismatch, restart, and unsafe automatic resume.

Job connection

How this becomes employable evidence

This capstone matches field integration work: connect a ROS graph, expose health to an operator/fleet service, reproduce QoS and callback failures, inhibit stale commands, and hand another engineer a launch file, architecture note, fault matrix, and decisive logs.

Relevant target roles

Robot HMI / Control & Monitoring Engineer
Robot Fleet Backend / Platform Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 05 interview drill

Interview questions: Multi-node watchdog demo and architecture note

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Design a watchdog for a 10 Hz robot-status stream. Explain the clock, thresholds, jitter allowance, state machine, false-positive test, simulated safe response, recovery rule, observability, and why ROS discovery or reliable QoS alone is insufficient.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why is a regularly arriving heartbeat with an unchanged old source timestamp unhealthy?

Model interview answer

Transport is active, but the physical evidence is stale; freshness must use the trustworthy source time and progress semantics.

Q2What should happen before a stale component returns to HEALTHY?

Model interview answer

A declared sequence of fresh, valid, correctly identified samples and any required reconfiguration/task reset must pass the recovery rule.

Q3What separates this ROS watchdog from a safety-rated emergency stop?

Model interview answer

It is ordinary application software over non-safety-rated compute/communication unless specifically engineered and certified otherwise; independent trusted safety hardware/controllers must enforce hazard limits.

Chapter starter artifact

Classify a stale heartbeat before commanding

One launch entry point starts a namespaced Jazzy graph whose typed telemetry, command action, QoS, callback groups, lifecycle readiness, stale-data watchdog, cancellation timeline, injected faults, and bounded recovery are independently inspectable.

week-05-aisle-seven-watchdog.mjsJavaScriptDownload starter

const mission = "aisle-seven-watchdog";
const expectedPeriodMs = 100;
const timeoutMs = 180;
const nowMs = 700;
const heartbeats = [
  { sequence: 21, sourceMs: 300 },
  { sequence: 22, sourceMs: 400 },
  { sequence: 23, sourceMs: 450 },
];
const latest = heartbeats.at(-1);
if (!latest || !heartbeats.every((item) =>
  Number.isFinite(item.sequence) && Number.isFinite(item.sourceMs)
)) throw new Error("invalid heartbeat");
const ageMs = nowMs - latest.sourceMs;
if (ageMs < 0) throw new Error("future heartbeat");
const sequenceAdvances = heartbeats.every(
  (item, index) => index === 0 || item.sequence > heartbeats[index - 1].sequence,
);
const health = ageMs > timeoutMs || !sequenceAdvances ? "STALE" : "HEALTHY";
const requestedCommand = 0.6;
const appliedCommand = health === "HEALTHY" ? requestedCommand : 0;
const recoverySamples = Math.ceil(timeoutMs / expectedPeriodMs) + 1;
console.log("mission=" + mission);
console.log("ageMs=" + ageMs);
console.log("health=" + health);
console.log("command=" + appliedCommand.toFixed(2));
console.log("recoverySamples=" + recoverySamples);
console.log("status=PASS");

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-05-aisle-seven-watchdog.mjs
Expected output: mission=aisle-seven-watchdog ageMs=250 health=STALE command=0.00 recoverySamples=3 status=PASS
Planted failure to diagnose: Replace the source timestamp with the local receipt time. Regularly replayed old heartbeats then look fresh and the simulated command remains enabled.