Chapter 20

Build a bounded language-to-ROS 2 task executor

Turn a natural-language request into an inspectable plan whose only physical operations are typed, validated robot skills. This chapter defines goals, preconditions, effects, and recovery branches; separates fresh world state from episodic memory; grounds every object reference in a timestamped perceived entity; executes through feedback, cancellation, postcondition checks, and bounded replanning; and closes uncertain outcomes with idempotency, compensation, safe human handoff, or abort. The result is a reproducible long-horizon simulator or tabletop demo with one correlated trace, not an unreviewable model-to-motor shortcut.

Before you start

Complete the ROS 2 action, lifecycle, behavior-tree, navigation or manipulation, cancellation, and recovery work from earlier chapters; bring one already working simulated skill such as navigate, detect, pick, place, or dock.
Bring a versioned simulator or harmless tabletop baseline with a documented reset, observable object identities and poses, a robot safe or idle state, and one known-good task that can be repeated without powered unsupervised hardware.
Be able to read a small JSON-like record, a state-transition table, and ROS 2 action feedback; distinguish a command being accepted, a controller returning success, and a physical postcondition being freshly observed.
Keep language-model output outside the actuator and safety boundary. Every exercise uses paper objects or simulation unless an already commissioned robot is operated under its approved supervision, workspace, speed, stop, and recovery procedures.

By the end

Translate a user request into a measurable terminal condition, constraints, typed subtasks, dependencies, and named recovery or handoff branches without emitting low-level motion from free-form text.
Specify a robot skill with typed inputs, units, frames, scene version, preconditions, result codes, feedback, timeout, cancellation, and an observable postcondition, then reject invalid calls before dispatch.
Maintain versioned current world state separately from append-only episodic history, retrieve only scoped evidence, and resolve stale or conflicting memories in favor of fresh trusted observation.
Ground plan references to stable perceived-entity records, reject ambiguity, re-observe before dispatch, and verify the expected physical effect after every skill.
Run an execute–monitor–re-observe loop that distinguishes progress loss, stale state, rejected goals, aborted actions, canceling goals, and failed postconditions before choosing a bounded retry or replan.
Handle timeout and outcome uncertainty with operation-specific idempotency, request identity, reconciliation, compensation, safe abort, and a complete human handoff package rather than automatic repetition.
Publish one long-horizon run whose goal, observations, plan revisions, skill calls, feedback, results, recoveries, interventions, timings, and final verified state can be reconstructed from a single trace.

The field story

Courier-17 Bounded Delivery

Courier-17 receives the Policy Passport from Week 19 and a natural-language request: move the blue inspection card to the marked tray. The model may propose a route through the task, but it cannot publish motion. Your mission is to turn that proposal into a small plan of typed skills whose inputs, frames, preconditions, timeouts, result codes, and observable postconditions can be inspected. A fresh entity record must anchor every noun, and the executor must distinguish a goal being accepted from the world actually changing. One ambiguous card or stale pose must stop the delivery before dispatch.

The courier's route becomes a recurring investigation. First you define the terminal condition and decompose the job. Then each skill receives a strict contract, while current world state stays separate from episodic memory. Before every action, Courier-17 re-observes the named entity; after every action, it verifies the expected effect. Timeouts, cancellation, idempotency, reconciliation, compensation, and human handoff close uncertain outcomes without blind repetition. The final trace must reconstruct every observation, plan revision, dispatch, feedback event, recovery, and terminal decision, giving Week 21 a real execution path to attack rather than a diagram that has never failed.

Why this chapter now: Week 19 produced a guarded policy proposal. It now needs deterministic orchestration that can reject ambiguity, monitor progress, and prove physical postconditions without granting the model actuator authority.
Ignore for now: Do not build a general autonomous agent, let free-form text call ROS interfaces directly, use memory as current truth, or retry an outcome-unknown operation blindly.
This unlocks: A traceable planner-to-skill executor that Week 21 can place behind independent action admission, threat, hazard, and evaluation gates.
Proof you will leave with: A measurable goal, typed skill schemas, world-state versions, entity grounding records, plan revisions, goal UUIDs, feedback and result events, postcondition observations, retry budgets, reconciliation evidence, and one complete correlated trace.

Environment contractrepository-supported Node.js 22.13.0 or newer runs the local state-machine starter with synthetic entities; the course mission remains in simulation or on paper and requires no model API, ROS graph, or powered robot.

Compatibility boundary: The starter demonstrates orchestration contracts only. A ROS 2 implementation must separately pin message/action definitions, clocks, frames, lifecycle behavior, cancellation semantics, simulator version, and skill adapters.
Smoke check: Run node week-20-courier-17.mjs; confirm the fresh entity dispatches, the planted stale entity is rejected, and the final mission state remains a safe handoff.
Contract reviewed: 2026-07-25
Runtime evidence: The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
Drift risk: medium

Today in the field story

One problem, then the next

Courier-17 begins with a sentence, but the mission board accepts only an observable terminal condition: the identified blue card is inside the marked tray, the gripper is clear, and the robot is idle before timeout. You decompose that outcome into perceive, approach, acquire, transport, place, verify, and safe-exit steps with dependencies and failure branches. The Week 19 policy can suggest this structure, yet every step remains a proposal until typed contracts and current world evidence make it executable.

Why now: A measurable goal prevents fluent language from hiding an undefined finish.
Ignore today: Ignore optimal planning and low-level trajectories.
Unlocks next: A bounded task graph with explicit terminal and recovery states.

Understand

Build the physical picture first

A robot plan is a checked dependency graph from current facts to a measurable goal, with each edge delegated to one bounded skill.

Begin with the physical end condition, not with a list of attractive verbs. “Tidy the bench” is ambiguous; “blue foam block B-17 is inside bin zone-2, the gripper is empty, the robot is parked, and no object left the marked workspace within 90 seconds” can be checked. Record forbidden actions, required human approvals, time limit, and abort state beside the goal. The planner may suggest a route to that state, but it may not weaken those constraints.

Decomposition replaces one broad request with skills that have named dependencies. A block cannot be placed until it has been located, grounded to one entity, reached, and grasped; parking depends on the gripper being empty. Independent perception or information-gathering steps may run in parallel, while physical steps that share a robot, object, or workspace usually need explicit ordering. Draw the dependency graph before translating it into a sequence, behavior tree, or planner representation.

Keep deliberation and execution as different components. A domain model names available types, predicates, and actions; current problem state names entities, facts, and the goal; a planner proposes a sequence; an executor checks current requirements and activates implementations. This PlanSys2-style separation makes a missing fact or failed precondition visible. It also prevents a language planner from turning “move closer” into unbounded wheel or joint commands.

A useful hierarchy becomes more concrete as it descends: mission, phase, skill, and controller goal. The mission may be “store block B-17”; phases may be perceive, acquire, deposit, and finish; skills may be detect, navigate, pick, place, and verify; controllers track bounded trajectories. Sequence and fallback branches should show what happens when a condition fails. Version every accepted plan so a later replan does not silently rewrite the history of what the executor actually attempted.

Words you need

Name each idea precisely

Goal condition: A set of observable facts that must be true for the task to count as complete, including required constraints and terminal robot state.
Physical example:
The foam block is visually confirmed inside the outlined bin, the gripper is empty, and the simulated arm is back at its named park pose.
Task decomposition: Breaking a broad objective into smaller skills whose inputs, order, ownership, and completion conditions can be inspected.
Physical example:
Store the block becomes observe the tray, resolve one block ID, approach, pick, observe grasp, place, observe the bin, and park.
Dependency: A fact or completed step that another step requires before it is eligible to run.
Physical example:
The place skill depends on a verified held-object fact, so it cannot run merely because the earlier pick command was sent.
Precondition: A predicate that must be freshly true immediately before a skill is dispatched.
Physical example:
Pick requires the chosen entity to be visible, inside the approved workspace, reachable, and not already reserved by another task.
Effect: The state change a skill is expected to cause and that later observation must confirm.
Physical example:
A successful place is expected to change held(B-17) into in_zone(B-17, zone-2) and gripper_empty.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalTask graphs, timeouts, and retries

An autonomous task is a state machine with measurable guards, not one long prompt.

t_deadline: latest allowed completion timeUnit: seconds (s)
N_retry: maximum retry countUnit: attempts
P(success): observed success frequencyUnit: probability

A grasp skill gets a 5 s timeout and at most 2 retries.
The maximum planned attempt time is 3×5 = 15 s, excluding recovery.
After each failure, re-observe and check a typed precondition before retrying; do not replay a stale command.

Programmer analogy

It resembles a workflow engine with typed APIs, except retries require fresh perception of a changed world.

One initial try plus three retries, each capped at 4 s, permits how much attempt time?

4 attempts × 4 s = 16 s.

If three required skills had independent success rates $0.9$ , $0.8$ , and $0.95$ , then

P(\mathrm{plan})\approx\prod_i p_i=(0.9)(0.8)(0.95)=0.684.

This rough estimate exposes chain fragility; real skill outcomes and recovery branches are not independent.

Decompose one storage request into an executable skill graph

The request is “put the blue block in bin B.” The current snapshot contains blue blocks B-17 and B-22, bin zones A and B, an empty gripper, a parked simulated arm, and a rule that the arm must never leave workspace W1.

Rewrite the request as terminal predicates: in_zone(B-17, B), gripper_empty, and arm_at(park), plus the invariant inside_workspace(arm, W1) and a 90-second deadline. Because two blue blocks exist, leave the target entity unresolved rather than guessing.
Add an information step clarify_target([B-17, B-22]). Suppose the operator selects B-17; bind that stable ID to the goal and record the selected observation snapshot instead of retaining the phrase “the blue block.”
List candidate skills and their contracts: observe scene, approach entity, pick entity, verify held object, place entity in zone, verify placement, and park arm. Do not include raw joint velocities or an unconstrained “fix failure” step.
Draw dependencies: approach needs a fresh pose; pick needs approach complete and reachable B-17; place needs held(B-17); park needs the gripper empty. The workspace invariant and active-stop condition guard every physical skill.
Add explicit branches: stale pose returns to observe, unreachable pose returns to replan or handoff, failed grasp allows one re-observation and one alternate-grasp attempt, ambiguous identity returns to clarification, and any safety rejection aborts physical execution.
Topologically order the graph into plan revision P1, validate every skill name and argument against the registry, and estimate timing from declared skill bounds. Reject P1 if its worst-case duration already exceeds 90 seconds.
Accept the plan only when another person can point from each terminal predicate backward to the observation or skill expected to establish it, and from every failure result to a bounded next state.

Result

Plan P1 contains one resolved entity, seven inspectable skills, declared dependencies and invariants, and finite clarification, re-observation, retry, handoff, and abort branches; no free-form text crosses the executor boundary.

What this proves

Good decomposition is not a longer to-do list: it makes state dependencies, physical authority, completion evidence, and every allowed failure branch explicit.

Physical examples

Where this appears in real life

Foam-block storage plan

A stationary simulated arm must put one named blue foam block from a pickup rectangle into a marked bin and return to park while a second similar block remains untouched.

Look for:

The plan first resolves the intended entity, orders pick before place, names the untouched-object constraint, and ends with freshly observed bin, gripper, and park conditions.

Cart delivery with a shared doorway

A simulated mobile robot must collect a light parcel at station A and deliver it to station B, but another robot can reserve the only narrow doorway.

Look for:

Navigation, parcel acquisition, doorway reservation, delivery, and release are separate skills; the plan contains wait, alternate-route, timeout, and handoff branches instead of assuming the doorway is free.

Hands-on exercise

Make the idea observable

Use index cards or a simulator with one arm or mobile base, two similar harmless objects, two target zones, and a written safe-state command. Keep actuators unpowered unless the platform and supervision are already approved.

Write one natural-language request, then replace it with three to five terminal predicates, one time limit, two invariants, and one safe abort condition that can all be observed.
Inventory only skills already implemented in the baseline. For each, write its owner, physical resource, input IDs, preconditions, expected effects, maximum duration, and failure results.
Draw a dependency graph and mark which steps can safely overlap, which must be serialized, and which require a fresh observation immediately before dispatch.
Insert one ambiguity, one stale-state case, one skill rejection, and one execution failure. Give each a finite clarify, re-observe, replan, handoff, or abort branch.
Walk the nominal graph with cards, changing a visible world-state sheet only after the corresponding postcondition card is checked.
Walk each injected failure without improvising a new action. Revise the plan model until every path reaches verified success, declared handoff, or safe abort in a bounded number of transitions.

Observe

Vague language accumulates at exactly the points where identity, ordering, ownership, and terminal evidence are missing; making those facts explicit removes many unsafe planner choices before execution.

Done when

The graph starts from a versioned snapshot, ends in measurable predicates, contains no raw actuator instruction, exposes all dependencies, and closes every injected failure without an unbounded loop.

Build today

Build an agent that converts a natural-language goal into inspectable ROS 2 actions and recovers from one failure.

Evidence to save

DONE when the learning log explains “Task decomposition and hierarchical planning” in five precise points and a checked example produces the predicted output.

Common mistakes

Catch the wrong mental model

Wrong

Treating natural-language subtasks such as “handle the object” or “fix the blockage” as executable skills.

Better

Replace each phrase with a registered bounded skill or an information-gathering, clarification, handoff, or abort state whose inputs and result are inspectable.

Wrong

Letting the high-level planner output joint positions, velocity commands, or ad hoc controller code.

Better

Limit the planner to registered skill calls; keep trajectory generation, command bounds, collision checks, and actuator control inside trusted deterministic components.

Wrong

Keeping one happy-path sequence after a precondition or effect becomes false.

Better

Model the failed predicate explicitly, return to the smallest safe re-observation or replan point, and cap every retry branch before execution begins.

Job connection

How this becomes employable evidence

Convert warehouse or service-robot requests into versioned mission graphs that call existing ROS 2 navigation and manipulation capabilities, preserve constraints across replans, and expose why a task clarified, waited, recovered, handed off, or aborted.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 20 interview drill

Interview questions: Task decomposition and hierarchical planning

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Decompose “bring me the red tote” for a mobile manipulator. Define terminal conditions, ambiguities, skill dependencies, planner-versus-executor ownership, and the branches you would require before allowing physical execution.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What is the first artifact to derive from “tidy the bench”?

Model interview answer

A measurable terminal condition with named entities, workspace and safety constraints, time and intervention limits, and an observable safe terminal robot state.

Q2Why separate planner, executor, and controller?

Model interview answer

The planner proposes registered skills, the executor validates current state and owns lifecycle and recovery, and controllers enforce bounded physical motion; one component cannot silently bypass the others' contracts.

Q3What makes a decomposition hierarchical rather than merely a flat checklist?

Model interview answer

Mission phases expand into typed skills and then bounded controller goals, with dependencies, invariants, and outcomes preserved across each level.

Chapter references

ROS 2 Design — ActionsPrimary ROS 2 action design for typed goal, feedback, and result sections; client-generated goal UUIDs; accepted, executing, canceling, succeeded, aborted, and canceled states; and the distinction between accepting a cancel request and reaching the canceled terminal state.
PlanSys2 — System DesignMaintainer architecture separating domain types, predicates, functions, and actions from current problem instances and goals, plan generation, and runtime execution by ROS 2 action performers with requirement checks.
BehaviorTree.CPP 4 — Reactive and Asynchronous BehaviorsMaintainer contract for non-blocking asynchronous action nodes, RUNNING versus terminal status, typed input ports, repeated monitoring, and prompt halt handling when a parent branch cancels work.
Nav2 — Detailed Behavior Tree WalkthroughOfficial worked behavior tree showing periodic replanning, contextual recovery around planning and path following, system-level recovery, and an explicit retry budget rather than unbounded repetition.
Google DeepMind — Gemini RoboticsPrimary product description of embodied reasoning, multi-step task decomposition, tool use, changing-environment response, and the separate roles of a high-level embodied-reasoning model and a vision-language-action model.