Chapter 05 · Build and debug a real ROS 2 system
Today in the field story
One problem, then the next
A timer inside the Aisle Seven Watchdog makes a synchronous request and waits forever because its completion callback shares the same mutually exclusive group. Reproduce the hang with a bounded timeout, diagram executor threads and groups, then repair the circular wait using asynchronous flow or compatible ownership. Measure callback duration, queue age, starvation, and shutdown after the repair.
- Why now
A healthy graph and compatible QoS cannot compensate for callbacks that never receive execution.
- Ignore today
Ignore adding threads as a default fix; first expose dependencies and shared state.
- Unlocks next
A schedulable watchdog whose timing and concurrency assumptions are reviewable.
Understand
Build the physical picture first
A node's callbacks are jobs waiting at a small workshop. The executor is the foreman deciding which ready job receives a worker thread. Callback groups are rules saying which jobs may overlap. Adding workers does not help if every job is locked inside one 'only one at a time' room.
Subscriptions, timers, service handlers, action handlers, and completion callbacks do not execute themselves. An executor waits for ready ROS entities and invokes their callbacks using one or more operating-system threads. A single-threaded executor runs one callback at a time. A multi-threaded executor can run callbacks concurrently only when their callback-group rules permit it. The middleware usually retains incoming data according to QoS until a callback takes it, so callback scheduling directly affects age, backlog, and deadline behavior.
A mutually exclusive callback group prevents callbacks in that group from overlapping. A reentrant group permits overlap, including another invocation of the same callback, so its code and state must be safe for concurrency. Different groups can execute in parallel under a multi-threaded executor. The default callback group is mutually exclusive; placing every entity in that group can make a multi-threaded executor behave like a single-threaded one. Thread count alone is therefore not a concurrency design.
A classic deadlock occurs when a callback makes a synchronous service call and waits, while the response-completion callback is assigned to the same mutually exclusive group. The waiting callback owns the group's only execution permission, so the completion callback cannot run and release it. Prefer asynchronous calls inside callbacks. When synchronous behavior is unavoidable, separate the waiting callback and client into compatible groups and prove the design under timeout and shutdown—but asynchronous state machines are usually easier to bound.
Concurrency can remove blocking and also create races. Two callbacks updating the same latest command, counter, or state machine may interleave. Protect shared state with a small clear ownership model, mutually exclusive group, lock, immutable message passing, or a single state-owning callback. Do not hold a lock while waiting for a ROS response. Record callback start/end, message source time, queue age, thread/group identity when useful, and deadline misses so scheduling failures are visible.
Every callback consumes part of a timing budget. A 20 Hz input arrives every 50 ms. If its callback consistently needs 80 ms on one execution path, backlog or dropped/overwritten samples are inevitable depending on QoS. Fixing it may mean doing less work in the callback, moving slow work to a bounded worker, lowering rate, using newest-data semantics, changing callback groups/executor, or improving the algorithm. Measure before adding threads, because parallel callbacks can increase contention and make ordering harder.
Words you need
Name each idea precisely
- Callback
A function invoked when a subscription, timer, service, action, or related event is ready.
Physical example:A battery callback runs when a new battery message is available.
- Executor
The ROS runtime component that waits for ready work and schedules callbacks on one or more threads.
Physical example:A single-threaded executor processes a timer before it can process a newly arrived status message.
- Mutually exclusive callback group
A group whose callbacks are never scheduled to overlap one another.
Physical example:Two callbacks that update one unprotected state machine can be kept sequential.
- Reentrant callback group
A group whose callbacks may run concurrently, including overlapping invocations.
Physical example:Independent image jobs may overlap only if their data and libraries are concurrency-safe.
- Deadlock
A state where work waits forever for an event or resource that cannot run or be released because of the waiting arrangement.
Physical example:A timer waits for a service response whose completion callback is blocked by that same timer's callback group.
- Starvation
A ready callback repeatedly fails to get execution time because other work monopolizes the available path.
Physical example:A long image callback delays a short watchdog timer until its deadline passes.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalMessage rates, queues, and latency
ROS nodes form a distributed timing system; rates and queue depth decide freshness.
- λ
- messages arriving each secondUnit: messages/s
- μ
- messages processed each secondUnit: messages/s
- latency
- receive time minus source timestampUnit: milliseconds (ms)
A camera publishes λ = 30 messages/s while a node processes μ = 20 messages/s.
The backlog grows by λ − μ = 10 messages each second.
A depth-5 queue fills in about 0.5 s; choose a QoS policy based on whether freshness or completeness matters.
ROS pub/sub resembles backend messaging, but an old robot message can command the wrong physical state.
Input is 50 Hz and processing is 40 Hz. How fast does backlog grow?
10 messages per second.
Serialized callback time is
A loop allows
so the serialized pass exceeds the budget by .
Find overload and explain a same-group deadlock
A telemetry subscription receives at 20 Hz. Its callback takes 80 ms. A 100 ms timer in the same default mutually exclusive group makes a synchronous service call whose done-callback inherits that group.
Calculate input period: T = 1/20 = 50 ms.
Compare callback duration with period: 80 ms - 50 ms = 30 ms of excess work per sample on a serial path.
After five arrivals, required callback work is 5 × 80 = 400 ms while only about 250 ms of arrival time has elapsed; the work demand exceeds the serial budget by 150 ms.
Trace the timer: it enters the default mutually exclusive group and waits synchronously for a service response.
Trace the response: its completion callback needs the same group, but that group cannot schedule it until the waiting timer returns.
Conclude that extra executor threads alone do not resolve this arrangement; use an asynchronous client flow or separate compatible callback groups with bounded waits.
The telemetry path is overloaded by 30 ms per sample, and the service path deadlocks because the waiting callback blocks the only group allowed to complete its wait.
Scheduling bugs are explainable with arrival periods, callback durations, group membership, and wait dependencies—not with 'ROS is slow'.
Physical examples
Where this appears in real life
One cashier and an age check
A cashier pauses a transaction and waits for a supervisor, but the supervisor can respond only through the same cashier station now occupied by the waiting transaction.
The wait cannot complete. A separate response path or asynchronous handoff is needed, mirroring the same-group synchronous service deadlock.
Slow camera processing beside heartbeat monitoring
An 80 ms image callback and a 5 ms heartbeat callback share a single-threaded executor while heartbeats arrive every 50 ms.
Heartbeat work can be late despite low CPU average. Separate groups/threads help only if shared state and timing are designed safely.
Hands-on exercise
Make the idea observable
Use a Python ROS 2 node with a timer, a numbered-message subscription, and a harmless add-two-numbers service. Log monotonic timestamps and callback names.
Run the timer and subscriber in the default callback group under a single-threaded executor; add an 80 ms simulated workload and measure timer jitter.
Increase the publish rate until callback work exceeds the available serial budget; record sequence gaps, age, and maximum timer delay.
Create a timer callback that makes a synchronous service call through a client in the same default group and reproduce the bounded test timeout rather than waiting forever.
Replace the synchronous wait with an asynchronous request and completion state; prove the timer remains responsive.
Under a multi-threaded executor, place independent callbacks into deliberate groups and repeat the measurement.
Introduce one shared counter without protection to understand the risk, then assign single ownership or a safe exclusion rule and document why it is correct.
A multi-threaded executor does not create useful parallelism when all callbacks remain in one mutually exclusive group. Asynchronous completion removes the circular wait but still needs timeout and shutdown handling.
Evidence contains the original jitter/age measurements, one reproducible deadlock timeout, a corrected asynchronous or properly grouped design, and a diagram showing callback group membership and shared-state ownership.
Build today
Build a Python/C++ telemetry, command, and watchdog system; then reproduce QoS, discovery, executor, lifecycle, and cancellation failures.
Evidence to save
DONE when a comparison table for “Executors, callback groups, concurrency, and deadlocks” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Switching to a multi-threaded executor and assuming every callback now runs in parallel.
Inspect callback groups; the default mutually exclusive group can serialize all entities despite several threads.
Making a synchronous service or action call from a callback without tracing its completion callback.
Prefer asynchronous calls or place dependencies in compatible groups, then enforce timeout and shutdown behavior.
Putting all work in reentrant groups to avoid blocking.
Use reentrancy only when overlapping access, library calls, and shared state are proven thread-safe.
Adding queue depth when callbacks are slower than arrivals.
Measure the service capacity and data age; reduce work/rate, bound or replace old samples, or redesign scheduling.
Job connection
How this becomes employable evidence
A robot appears to freeze only when diagnostics run during navigation. The engineer correlates callback durations, group assignments, executor threads, service waits, and message age; then removes the circular wait and adds a regression test for heartbeat deadline.
Relevant target roles
- Robot Fleet Backend / Platform Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 05 interview drill
Interview questions: Executors, callback groups, concurrency, and deadlocks
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Why can a multi-threaded executor still behave serially, and how can a synchronous service call from a callback deadlock? Draw the wait dependency and propose a bounded fix.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why might four executor threads still execute a node's callbacks one at a time?
If all entities use the same mutually exclusive callback group, that group permits only one callback at a time.
Q2What creates the same-group synchronous service deadlock?
The waiting callback occupies the mutually exclusive group while the response completion callback needs that same group to run and satisfy the wait.
Q3What should you measure before deciding to add threads?
Arrival periods, callback durations, message age, queue/loss behavior, deadline misses, wait dependencies, and shared-state contention.