Chapter 05 · Build and debug a real ROS 2 system
Today in the field story
One problem, then the next
Telemetry is typed, yet the Aisle Seven Watchdog is silent across two containers. Inspect domain, discovery, middleware, network path, endpoint type, and requested/offered QoS before changing code. Compare best-effort freshness with reliable delivery under bounded queues, then plant one incompatibility and one domain mismatch so their different symptoms remain available as regression evidence.
- Why now
Communication policy and discovery must be observable before silence can be classified correctly.
- Ignore today
Ignore security claims; a domain ID is not authentication or encryption.
- Unlocks next
A defensible QoS table and network diagnosis for sensors, state, and commands.
Understand
Build the physical picture first
QoS is the delivery agreement attached to a message channel. A live camera feed, an emergency state, and a static map are all 'data', but one values newest frames, one values timely state changes, and one must be available to late joiners. They should not share a delivery policy by habit.
ROS 2 uses a middleware layer, commonly based on DDS, to discover participants and transport data. Nodes in the same ROS domain announce endpoints and establish connections when topic names, interface types, and Quality of Service policies are compatible. Discovery failure can come from a different ROS_DOMAIN_ID, blocked multicast or ports, container/network isolation, middleware configuration, or incompatible security settings. Seeing two machines on the same Wi-Fi is not proof that their ROS endpoints can discover one another.
A QoS profile combines policies. Reliability may be best effort or reliable. History and depth decide how many samples are retained when using keep-last. Durability decides whether a late-joining subscriber can receive retained samples from a transient-local publisher or only future samples from a volatile one. Deadline expresses an expected maximum interval, lifespan limits how long data remains useful, and liveliness helps detect whether a publisher is considered alive. These policies describe middleware behavior; application-level semantic checks and safe-state logic are still required.
Reliable does not mean 'always safest'. On a lossy link, retransmitting every old camera frame can fill queues and increase age while the robot needs the newest view. Best effort with a small depth is often appropriate for high-rate sensors when occasional loss is acceptable. A slowly changing map or configuration may need transient-local durability so a subscriber that starts later receives the latest retained state. Command and safety-related data need a hazard-specific design, acknowledgement or state feedback when required, and independent safety controls—not a blind reliable flag.
Compatibility follows a request-versus-offered model. For reliability, a reliable publisher can satisfy a best-effort or reliable subscriber, but a best-effort publisher cannot satisfy a subscriber requesting reliable delivery. Durability has a similar direction: a volatile publisher cannot provide historical data requested by a transient-local subscriber. A connection can therefore be visible as two endpoints on the same topic yet deliver nothing because the policies do not match. Inspect offered/requested QoS rather than rewriting callbacks.
Choose QoS from measured physical needs. Write the expected source rate, maximum useful age, tolerated loss, late-join behavior, memory bound, reconnect behavior, and consequence of missing data. Then inject loss, delay, restart, and incompatible profiles. Topic rate and delivery percentage are useful measurements, but they do not reveal whether received data was fresh enough, in order, or semantically valid.
Words you need
Name each idea precisely
- DDS discovery
The distributed middleware process by which ROS endpoints announce themselves and find compatible peers.
Physical example:A laptop dashboard discovers the rover's battery publisher only when network and domain settings permit it.
- Reliability
A QoS policy choosing best-effort delivery or middleware retry behavior for reliable delivery.
Physical example:A live camera may prefer fresh best-effort frames; a low-rate state transition may require stronger delivery evidence.
- History and depth
Policies controlling whether samples are kept and the bounded number retained for keep-last.
Physical example:Depth one lets a slow display skip old pose samples and work on the newest available pose.
- Durability
A policy controlling whether data can be retained for subscribers that join after publication.
Physical example:A transient-local map publisher can provide its last map to a visualization started later.
- Deadline
The expected maximum time between successive data samples for a connection.
Physical example:A 10 Hz heartbeat expects samples around 100 ms apart and can report a missed 150 ms gap against a 120 ms deadline.
- Liveliness
A middleware indication of whether a publisher is considered alive under a lease policy.
Physical example:A node can lose its liveliness lease even before an operator notices its process is gone.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalMessage rates, queues, and latency
ROS nodes form a distributed timing system; rates and queue depth decide freshness.
- λ
- messages arriving each secondUnit: messages/s
- μ
- messages processed each secondUnit: messages/s
- latency
- receive time minus source timestampUnit: milliseconds (ms)
A camera publishes λ = 30 messages/s while a node processes μ = 20 messages/s.
The backlog grows by λ − μ = 10 messages each second.
A depth-5 queue fills in about 0.5 s; choose a QoS policy based on whether freshness or completeness matters.
ROS pub/sub resembles backend messaging, but an old robot message can command the wrong physical state.
Input is 50 Hz and processing is 40 Hz. How fast does backlog grow?
10 messages per second.
For ,
A queue depth of spans at most
at the nominal rate.
Diagnose a silent subscriber and a missed deadline
A publisher offers best-effort reliability at 10 Hz. A subscriber requests reliable delivery and a 120 ms deadline. The publisher sends 100 samples; after compatibility is corrected, 96 arrive, and one measured gap is 150 ms.
Calculate nominal period: T = 1/10 = 0.1 s = 100 ms.
Compare reliability: best-effort offered cannot satisfy reliable requested, so the original pair is incompatible and exchanges no data.
Change the subscriber to request best effort only if the application's loss tolerance permits it; do not change policy merely to make the warning disappear.
Calculate delivery fraction after connection: 96/100 = 0.96 = 96%.
Compare the 150 ms observed gap with the 120 ms deadline; the gap misses the deadline by 30 ms.
Report both results: 96% sample delivery and one missed deadline. Neither number alone proves the received samples were fresh or safe to use.
The initial silence is explained by reliability incompatibility. After a justified compatible profile is selected, delivery is 96%, but a 150 ms interval violates the 120 ms deadline.
QoS diagnosis starts with compatibility, then measures rate, gaps, age, loss, and reconnect behavior against the data's real purpose.
Physical examples
Where this appears in real life
Camera stream versus static map
A camera sends 30 frames/s over variable Wi-Fi; a map changes once during startup and a viewer may launch five minutes later.
The camera values current frames and bounded queues, while the map values a retained latest sample for a late joiner.
Two robots in a lab
Student A's laptop controls robot A and Student B's laptop controls robot B on the same network.
Separate ROS domain IDs prevent accidental discovery, but domain separation is operational isolation rather than authentication or a complete security boundary.
Hands-on exercise
Make the idea observable
Use two local ROS 2 terminals and a harmless numbered-message publisher/subscriber. Save the original domain and middleware environment before changing it.
Run the endpoints with the same domain, topic type, and compatible best-effort profiles; record endpoint information and received sequence numbers.
Change only the subscriber to reliable while the publisher remains best effort; inspect the compatibility warning and prove messages stop.
Restore a justified compatible pair, use keep-last depth one, slow the subscriber, and observe that current data replaces queued old samples.
Run both endpoints with different temporary
ROS_DOMAIN_IDvalues and prove they do not discover each other; then restore the original environment.Restart a volatile publisher and a late subscriber, then repeat with a transient-local latest-state example; record what a late joiner receives.
Create a table for camera, map, heartbeat, and operator-command data listing reliability, durability, depth, deadline/age limit, loss tolerance, and rationale.
Identical topic names and types are insufficient when QoS or domains differ. A connection that works after changing QoS still needs a reason tied to freshness, loss, and task risk.
The evidence includes one compatibility failure, one domain discovery failure, one late-join comparison, measured sequence/rate behavior, and a defensible QoS table rather than a universal 'reliable' default.
Build today
Build a Python/C++ telemetry, command, and watchdog system; then reproduce QoS, discovery, executor, lifecycle, and cancellation failures.
Evidence to save
DONE when the integrated “QoS profiles and DDS discovery across networks” path is observable, cancelable, and leaves the prior baseline reproducible.
Common mistakes
Catch the wrong mental model
Setting every topic to reliable because the word sounds safer.
Choose policies from freshness, loss tolerance, late-join needs, memory bounds, and task hazards; test under the real link conditions.
Treating ROS_DOMAIN_ID as authentication or complete network security.
Use it for discovery isolation, then apply ROS security and network controls appropriate to the threat model.
Checking only topic name and message type when no data arrives.
Inspect domain, discovery path, middleware, endpoint counts, and requested/offered QoS compatibility.
Using a large queue to hide a slow subscriber.
Measure callback capacity and data age; bound the queue and define overflow or newest-data behavior explicitly.
Job connection
How this becomes employable evidence
At a warehouse, a dashboard sees robot status but not lidar. The engineer compares domains, network reachability, middleware choice, endpoint type, and offered/requested QoS; then chooses a sensor profile that keeps frames fresh rather than creating a growing reliable backlog.
Relevant target roles
- Robot HMI / Control & Monitoring Engineer
- Robot Fleet Backend / Platform Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 05 interview drill
Interview questions: QoS profiles and DDS discovery across networks
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A best-effort lidar publisher and reliable subscriber appear on the same topic but exchange no scans. Explain the compatibility rule, a justified fix, and the measurements needed before calling the stream healthy.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Can a best-effort publisher satisfy a subscriber that requests reliable delivery?
No. The publisher does not offer the minimum reliability requested by that subscriber.
Q2Why can transient-local durability help a map viewer?
A compatible late-joining viewer can receive the retained latest map instead of waiting for it to be republished.
Q3What does a 99% delivery rate fail to tell you?
It does not show sample age, ordering, deadline misses, semantic validity, which sample was lost, or whether the missing data caused an unsafe task state.