Chapter 05

Build and debug a real ROS 2 system

Learn ROS 2 as a distributed runtime rather than as a collection of commands. By the end of the week, you will be able to split a robot application into well-named nodes, choose a topic, service, or action deliberately, design typed messages, match QoS to the physical data, reason about callback scheduling, launch a reproducible multi-node system, and prove that a watchdog detects stale components without issuing unsafe guesses.

Before you start

Use a Linux terminal, read a process log, and distinguish source code, a built executable, and a running process.
Read frequency in hertz and calculate period with T = 1/f; compare timestamps in seconds and milliseconds.
Write a small Python function or class and recognize that concurrent callbacks may touch shared state in an unexpected order.
Have a supported ROS 2 environment or container available; use simulation and harmless numeric messages rather than powered hardware for this chapter.

By the end

Explain the difference between a ROS 2 distribution, workspace, package, executable, node, and live graph.
Design a topic message with explicit meaning, type, unit, frame, timestamp, validity, and bounded data where appropriate.
Choose between topics, services, and actions from the lifetime and cancellation needs of the physical task.
Select compatible QoS for sensor, state, and command data and diagnose a discovery or compatibility failure from evidence.
Explain how executors and callback groups schedule work, then reproduce and remove one starvation or deadlock condition.
Use namespaces, parameters, lifecycle states, composition boundaries, launch files, and CLI introspection to make bring-up repeatable.
Ship a telemetry, command, and watchdog graph with measured freshness limits, fault injection, safe simulated behavior, and an architecture note.

The field story

The Aisle Seven Watchdog

A warehouse simulation loses updates from the robot assigned to Aisle Seven. The fleet screen still shows the node name, a reliable topic is configured, and the last numeric value looks normal, yet nobody can tell whether the data is fresh or whether cancellation reached the command path. You inherit a small ROS 2 system and must separate installed packages from the live graph, define typed interfaces, choose communication lifetimes, expose QoS and discovery evidence, and make callback scheduling failures reproducible.

The Aisle Seven Watchdog becomes the chapter’s continuing distributed-system story. Telemetry, command action, watchdog, and operator status are added in deliberate stages to one namespaced graph. Every interface carries physical meaning and time; every long task has terminal and cancellation semantics; every retry and recovery is bounded. The course path remains ROS 2 Jazzy. Lyrical is compatibility-only until a separate full-stack matrix passes, so no learner should mix tutorial commands or binaries midway through this mission.

Why this chapter now: The deterministic gateway must now cooperate with concurrent processes and machines without losing physical meaning, freshness, ownership, or terminal outcomes.
Ignore for now: Ignore Gazebo physics, robot geometry, navigation, SROS2 configuration, and powered commands. Use harmless numeric messages and simulated command inhibition.
This unlocks: URDF/tf2 integration, ros2_control, Gazebo bridges, Nav2 actions, fleet health, and production operator traces.
Proof you will leave with: Save the intended and observed graph, interface definitions, QoS/discovery matrix, cancellation trace, deadlock reproduction and repair, lifecycle launch evidence, watchdog fault table, starter output, and architecture note.

Environment contractUbuntu 24.04 with ROS 2 Jazzy is the golden course path. Use one sourced Jazzy underlay and one clean workspace overlay; this chapter does not require Gazebo. The dependency-free starter runs with repository-supported Node.js 22.13.0 or newer.

Compatibility boundary: ROS 2 Lyrical is an isolated compatibility target only. Do not mix Lyrical, Kilted, rolling, or Jazzy packages in one workspace; promote only after the same interfaces, builds, tests, QoS, timing, and behavior pass.
Smoke check: Run source /opt/ros/jazzy/setup.bash, confirm ROS_DISTRO=jazzy, run ros2 doctor --report, and verify the installed demo node package before creating the chapter workspace.
Contract reviewed: 2026-07-25
Runtime evidence: The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
Drift risk: high

Today in the field story

One problem, then the next

The Aisle Seven Watchdog starts by distinguishing its Jazzy installation, workspace, packages, executables, processes, nodes, and current graph. Draw the planned telemetry, command, monitor, and operator nodes, then stop one process and compare runtime evidence with files still present on disk. Record the sourced underlay and overlay so a second terminal cannot silently use another environment.

Why now: A distributed fault cannot be diagnosed while build artifacts and live participants are treated as the same layer.
Ignore today: Ignore Lyrical migration beyond a separate compatibility note.
Unlocks next: A stable graph on which message and task contracts can be designed.

Understand

Build the physical picture first

Think of a robot application as a school at lunchtime. The school building is the installed ROS distribution, a classroom cupboard is a package, a printed job card is an executable, each person currently doing one job is a node, and the live pattern of people exchanging labelled notes is the ROS graph.

ROS 2 is not a robot brain and it is not a complete operating system. It is a set of libraries, interface rules, build tools, command-line tools, and middleware that helps independently running pieces of robot software find one another and exchange typed data. A camera driver, localization estimator, mission manager, and operator display can be written separately yet participate in one observable system. That separation is useful only when every boundary has a clear name, data contract, timing expectation, and failure behavior.

A ROS distribution is a tested family of ROS packages released together. As of this course audit, Lyrical Luth is the current long-term-support release from May 2026 and is supported through May 2031; Jazzy Jalisco remains an older supported LTS baseline. Choose the distribution that matches the team's operating system, vendor drivers, simulator, and deployment image instead of mixing package versions casually. A workspace is a directory in which packages are developed and built. An installed ROS distribution is commonly the underlay; a sourced development workspace becomes an overlay that can replace or add packages for that terminal.

A package is the unit used to organize, build, test, and share related code or interfaces. It may contain several executable programs. A node is a running ROS participant created by an executable, and one process may contain one node or several composed nodes. The live ROS graph contains the node names and their topic, service, and action connections at this moment. Deleting a source file does not instantly erase a running node, and merely having a package on disk does not place anything in the graph.

Nodes discover one another through the selected ROS middleware, normally across processes and possibly across machines. Discovery is distributed: there is no required central ROS master. Participants need a compatible ROS domain, network path, middleware configuration, interface type, and QoS. The graph can change as processes start, stop, or reconnect, so a diagram in a README is a design claim while ros2 node list, ros2 node info, and related tools are runtime evidence. Strong ROS debugging compares the intended graph with the observed graph before changing application code.

The basic development loop is source the chosen distribution, create or enter a workspace, add packages under src, resolve declared dependencies, build with colcon, source the resulting overlay, run executables, and inspect the graph. Sourcing affects the current shell, which is why one terminal can find a package while another cannot. Record the distribution, operating system, middleware implementation, workspace revision, build command, and environment variables in the project instructions so another engineer can reproduce the same graph.

Words you need

Name each idea precisely

ROS distribution: A named, versioned release of compatible ROS 2 packages with a defined support period and target platforms.
Physical example:
A field robot image may standardize on Lyrical while an older vendor driver is qualified only on Jazzy.
Workspace: A directory containing ROS packages plus the build, install, and log outputs produced for them.
Physical example:
The telemetry and watchdog packages for one prototype can live under the same robotshift_ws/src directory.
Package: The build and distribution unit that owns related code, interfaces, tests, configuration, and metadata.
Physical example:
A robotshift_monitoring package can provide both a heartbeat node and its launch file.
Node: A currently running participant that performs a logical job and communicates through ROS interfaces.
Physical example:
A battery node reads voltage while a separate display node presents it to an operator.
ROS graph: The changing runtime network of nodes and their topic, service, and action connections.
Physical example:
When the lidar driver stops, its node and publisher disappear from the observed graph.
Underlay and overlay: An underlay supplies an existing ROS environment; a sourced overlay adds or replaces packages built in a workspace.
Physical example:
The installed Lyrical desktop is the underlay and your newly built watchdog package is in the overlay.

Visual model

See the relationship

Swipe the technical canvas horizontally on a small screen.

Day 29 · Concept — Interfaces, lifecycle, and observations match the contract. Measured anchor: 100 % healthy.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalMessage rates, queues, and latency

ROS nodes form a distributed timing system; rates and queue depth decide freshness.

λ: messages arriving each secondUnit: messages/s
μ: messages processed each secondUnit: messages/s
latency: receive time minus source timestampUnit: milliseconds (ms)

A camera publishes λ = 30 messages/s while a node processes μ = 20 messages/s.
The backlog grows by λ − μ = 10 messages each second.
A depth-5 queue fills in about 0.5 s; choose a QoS policy based on whether freshness or completeness matters.

Programmer analogy

ROS pub/sub resembles backend messaging, but an old robot message can command the wrong physical state.

Input is 50 Hz and processing is 40 Hz. How fast does backlog grow?

10 messages per second.

The build-matrix pass rate is

P=\frac{7}{8}\times100\%=87.5\%.

The remaining $1$ failed configuration still blocks compatibility even though the support-date difference is approximately $2031-2029=2$ years.

Separate files on disk from the live four-node graph

A workspace contains packages robotshift_sensing, robotshift_control, and robotshift_monitoring. At runtime, telemetry publishes at 20 Hz to a filter and a watchdog; a command node publishes to a simulated actuator.

List the static package units: sensing, control, and monitoring. Do not count them as nodes.
List the runtime nodes: /telemetry, /filter, /watchdog, and /sim_actuator; the graph therefore has four named participants.
Draw directed topic edges /telemetry→/filter, /telemetry→/watchdog, and /command→/sim_actuator, labelling the publisher and subscriber on each side.
Calculate the expected telemetry period: T = 1/20 Hz = 0.05 s = 50 ms.
Mark /watchdog stopped and redraw the observed graph. The monitoring package still exists on disk, but its running node and subscription no longer exist.
Compare the design drawing with ros2 node list, ros2 node info /telemetry, and ros2 topic info /telemetry; record any mismatch as a runtime fact.

Result

The design contains three packages, while the healthy runtime contains four nodes and three topic connections; a 20 Hz stream should produce samples about 50 ms apart.

What this proves

Package structure answers where software is built; the graph answers what is running and connected now. Never substitute one for the other.

Physical examples

Where this appears in real life

Warehouse mobile robot

A base driver reports wheel state, a localization node estimates the robot pose, a mission node requests movement, and a tablet shows progress.

Look for:

Four logical jobs can run in separate processes or machines. The graph should expose named typed connections, while the workspace/package layout merely explains where the programs came from.

Smart home as a tiny robot network

A temperature sensor publishes readings, a controller decides whether cooling is needed, and a dashboard displays the latest state.

Look for:

Installing all three programs does not create a live system. Only running participants with matching interfaces and discovery settings appear in the graph.

Hands-on exercise

Make the idea observable

Use a supported local ROS 2 installation or container with the official demo nodes. Keep this exercise on one computer and use harmless text messages.

Open a clean terminal, source the selected ROS distribution, and record ROS_DISTRO, RMW_IMPLEMENTATION, and ROS_DOMAIN_ID if set.
Create ~/robotshift_ws/src, enter the workspace, and create one empty Python package named robotshift_graph with its dependencies declared in package metadata.
Build the workspace with colcon, source its install overlay in a new terminal, and prove the package is discoverable there but not in an unsourced clean shell.
Run the official talker and listener demo nodes in separate sourced terminals.
Use node, topic, and interface CLI commands to record each node name, the topic name, publisher/subscriber counts, and message type.
Stop the listener, repeat the graph inspection, and annotate exactly what disappeared and what remained.

Observe

Sourcing controls which packages the shell can resolve, while starting and stopping processes changes the live graph. The talker can remain present even with no listener.

Done when

A short evidence note contains the selected distribution and middleware, successful workspace build, intended graph, observed healthy graph, observed graph after one node stops, and the exact CLI outputs that support each claim.

Build today

Build a Python/C++ telemetry, command, and watchdog system; then reproduce QoS, discovery, executor, lifecycle, and cancellation failures.

Evidence to save

DONE when the learning log explains “ROS 2 distributions: Jazzy baseline, Lyrical LTS, workspaces, packages, and graph” in five precise points and a checked example produces the predicted output.

Common mistakes

Catch the wrong mental model

Wrong

Calling a package, executable, process, and node the same thing.

Better

State which layer you mean: package on disk, executable built from it, operating-system process, or ROS node participating in the graph.

Wrong

Building an overlay and forgetting to source it in the terminal that launches the node.

Better

Source the distribution and workspace deliberately, then record the environment in the launch or runbook.

Wrong

Assuming a saved graph diagram proves current connectivity.

Better

Inspect the live graph and compare it with the diagram under a named test condition.

Wrong

Mixing packages from different distributions because their names look identical.

Better

Use one qualified distribution and operating-system baseline unless compatibility is explicitly tested.

Job connection

How this becomes employable evidence

During customer-site bring-up, a package builds correctly but its driver node is absent. The engineer checks which workspace was sourced, whether the process actually launched, its node name and namespace, dependency/version compatibility, and the live graph before touching sensor wiring.

Relevant target roles

Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 05 interview drill

Interview questions: ROS 2 distributions: Jazzy baseline, Lyrical LTS, workspaces, packages, and graph

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A teammate says, 'The package is installed, so the node must be running.' Explain why that is false and give a six-step evidence-first diagnosis.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What changes when a node process stops but its source package remains installed?

Model interview answer

The package stays on disk, but the node and its runtime interfaces disappear from the live graph after discovery updates.

Q2Why can two terminals on the same computer find different ROS packages?

Model interview answer

Each shell has its own sourced underlay/overlay environment and related variables.

Q3What is the first evidence that a planned publisher-subscriber connection exists?

Model interview answer

The live graph shows the intended node names, topic, matching interface type, endpoint counts, and compatible QoS; data and timestamps then prove messages are actually flowing.

Chapter references

ROS 2 documentation — Lyrical Luth releaseDay 29: current distribution choice. Lyrical was released in May 2026, is an LTS release, and is supported until May 2031; the page also records executor and observability changes introduced in Lyrical.
ROS 2 Lyrical — TutorialsDays 29, 30, and 34: environment setup, workspaces, packages, nodes, topics, parameters, launch, logging, composition, lifecycle demos, and command-line introspection.
ROS 2 Lyrical — Topics vs Services vs ActionsDays 30 and 31: choosing continuous streams, short request-response work, or long-running preemptible work with feedback.
ROS 2 documentation — Quality of Service settingsDay 32: history, depth, reliability, durability, deadline, lifespan, liveliness, predefined profiles, and request-versus-offered compatibility.
ROS 2 documentation — ExecutorsDays 33–35: executor scheduling, callback groups, multi-threaded behavior, wait sets, timing limitations, and the Lyrical Callback Group Events Executor.