Chapter 20 · Build a bounded language-to-ROS 2 task executor
Today in the field story
One problem, then the next
Courier-17 starts moving through the plan, but execution is not a one-way checklist. Each running skill emits feedback; the supervisor checks progress, deadlines, state freshness, and cancellation. After a terminal result, the camera or simulator verifies the intended effect before the next step. A blocked route causes one bounded re-observation and replan, not an infinite loop. The trace distinguishes planner revision, skill retry, and unchanged physical state so a later evaluator can reconstruct why the courier continued or stopped.
- Why now
Long tasks fail between steps unless execution repeatedly closes the observation loop.
- Ignore today
Do not optimize success rate or add broad recovery libraries yet.
- Unlocks next
An execute-monitor-re-observe loop with bounded replanning.
Understand
Build the physical picture first
Long-horizon execution is a repeated contract: dispatch one eligible skill, monitor it, observe its effect, then plan from the world that actually exists.
Track each skill attempt through explicit states such as ready, dispatched, accepted, running, canceling, verifying, succeeded, failed, outcome-unknown, and handed-off. Store entry time, request and goal IDs, plan revision, input scene version, and owner at every transition. A ROS action can end succeeded, aborted, or canceled, but the orchestrator adds a verifying state because transport completion and physical completion are different claims.
Monitor evidence that can change the decision: goal status, progress values, progress age, robot mode, stop state, localization or scene freshness, resource lease, and invariant violations. Define expected progress for the skill rather than accepting any changing number. For navigation it might be bounded decrease in distance over a window; for a pick it might be phase changes followed by held-object evidence. Feedback silence and lack of physical progress are separate faults.
After every action, re-observe the predicates on which the remaining plan depends. Compare expected effects with current facts and classify the mismatch before choosing recovery. A transient stale camera may justify reacquisition; a moved target invalidates a plan suffix; a blocked aisle may require a different path; a lost localization state can invalidate every pose-dependent step; an unknown active goal may require reconciliation before any retry.
Recovery has a budget and a reason. Contextual recovery addresses a specific failed skill, such as refreshing a costmap before recomputing a path. System recovery addresses broader state, such as relocalization. Replanning uses a new snapshot and increments the plan revision; repeating the identical call without changing its failed precondition is not replanning. Exhausted budget, violated invariant, increasing risk, or unreliable state ends in safe abort or human handoff.
Words you need
Name each idea precisely
- Execution monitor
The component that correlates an active skill with status, feedback, timers, current invariants, and expected progress.
Physical example:It detects that a navigation goal remains executing while distance and pose have not changed for five seconds.
- Progress predicate
A skill-specific, time-bounded statement describing observable movement toward completion rather than mere process activity.
Physical example:Remaining path distance must fall by at least 0.1 m within four seconds while localization stays fresh.
- Postcondition
The fresh world-state fact required after execution before a skill may be treated as physically successful.
Physical example:After place, the intended block is inside the target polygon and the gripper is observed empty.
- Replanning
Generating a new plan revision from updated state after assumptions, preconditions, or effects invalidate the previous plan.
Physical example:A trolley blocks aisle A, so snapshot 52 creates plan P3 through aisle B instead of resending P2's path.
- Recovery budget
A finite per-skill and per-task limit on retries, recovery actions, elapsed time, or risk before handoff or abort.
Physical example:One sensor reacquisition and one replan are allowed; a second failure parks the robot and requests an operator.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalTask graphs, timeouts, and retries
An autonomous task is a state machine with measurable guards, not one long prompt.
- t_deadline
- latest allowed completion timeUnit: seconds (s)
- N_retry
- maximum retry countUnit: attempts
- P(success)
- observed success frequencyUnit: probability
A grasp skill gets a 5 s timeout and at most 2 retries.
The maximum planned attempt time is 3×5 = 15 s, excluding recovery.
After each failure, re-observe and check a typed precondition before retrying; do not replay a stale command.
It resembles a workflow engine with typed APIs, except retries require fresh perception of a changed world.
One initial try plus three retries, each capped at 4 s, permits how much attempt time?
4 attempts × 4 s = 16 s.
Under the stated independence assumption, all five stages succeed with probability
Recover a delivery plan after a new obstacle
Plan P7 for a simulated cart is navigate(A) → acquire parcel Q3 → navigate(B) → release Q3 → verify delivery → park. Navigation permits one contextual clear-and-replan recovery and one whole-task replan within a 120-second task deadline.
Dispatch
navigate(A)from snapshot S40 with request R18 and goal UUID G18. Record accepted and running transitions, localization age, initial distance, deadline, and progress-window rule.Insert a trolley. Status remains running and feedback arrives, but distance changes by less than 0.1 m for four seconds. The monitor raises
NO_PROGRESS; it does not wait for the full task deadline or call this success because feedback is present.Request cancellation for G18, enter canceling, and wait for terminal canceled status while verifying zero commanded motion. If the goal remains outcome-unknown, do not send a conflicting goal.
Run the allowed contextual recovery: refresh obstacle state and compute a new path. Snapshot S41 now contains the trolley and marks the original path invalid; the recovery budget changes from two available units to one.
Create plan P8 from S41. Keep the unchanged parcel and terminal predicates, replace only the invalid navigation suffix, record the relation
supersedes P7, and verify worst-case remaining time still fits the task deadline.Execute the alternate route while monitoring goal status, feedback age, progress, localization, authority, and stop state. After navigation succeeds, verify measured pose at station A before allowing parcel acquisition.
Complete the remaining skills with a postcondition observation after each. Report recovered success only if Q3 is freshly observed in B, the cart is parked, no invariant failed, and the trace shows one cancellation, one replan, and no hidden intervention.
The task survives a real state change by canceling one correlated action, updating the scene, revising only the affected plan, spending a visible recovery budget, and verifying each downstream effect.
Recovery is evidence-driven state change plus a bounded new decision; an identical retry from the same failed assumptions is only repetition.
Physical examples
Where this appears in real life
Blocked simulated aisle
A cart begins a navigation skill, then a trolley appears on its path. The action still publishes feedback, but distance stops decreasing within the declared window.
The monitor detects lost progress, cancels the exact goal, waits for a terminal state, observes the obstacle, creates a new snapshot and plan revision, and spends one recovery unit.
Successful command, empty gripper
A pick action reports succeeded after closing the gripper, yet the fresh scene shows the object still on the tray and the gripper opening consistent with no object.
The orchestrator enters verifying, returns POSTCONDITION_FAILED, updates the world state, and chooses at most the declared alternate-grasp recovery instead of running place.
Hands-on exercise
Make the idea observable
Use a simulator or a paper state machine with one multi-step delivery or storage task. Prepare injectors for lost progress, stale observation, failed postcondition, and cancellation.
Define the per-attempt states, allowed transitions, required fields, progress predicate, feedback-silence limit, total timeout, postcondition, and recovery budget for two skills.
Run the nominal task and produce a trace showing each dispatch, goal acceptance, feedback, terminal result, verifying observation, and next-step eligibility.
Inject feedback silence while physical progress continues; classify communication freshness separately from physical progress and follow the declared degraded or cancel rule.
Inject fresh feedback with no progress; cancel the exact action, wait for its terminal state, and prove another conflicting goal does not begin early.
Inject a successful action result with a failed physical postcondition; update world state and ensure the dependent skill remains blocked.
Change the scene, increment its version, create a new plan revision, and record which prior steps remain valid, which suffix changed, and which recovery budget was consumed.
Exhaust the budget in a final run and verify the system enters the written safe abort or handoff state without another automatic attempt.
Status, feedback, progress, state freshness, and postconditions answer different questions; failures become diagnosable only when the trace retains all of them and the exact transition they caused.
Nominal, no-progress, stale-feedback, failed-postcondition, and exhausted-budget cases reach their declared terminal states with no overlapping conflicting action and no hidden retry.
Build today
Build an agent that converts a natural-language goal into inspectable ROS 2 actions and recovers from one failure.
Evidence to save
DONE when a comparison table for “Execute, monitor, re-observe, and replan long-horizon tasks” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Calling a skill healthy because feedback messages continue to arrive.
Monitor feedback freshness and a separate physical progress predicate, plus current invariants and postconditions.
Retrying the same arguments after a failed precondition or unchanged obstacle.
Re-observe, identify the invalid assumption, change relevant state or plan, and consume a finite recovery budget before another attempt.
Dispatching a replacement action immediately after sending a cancel request.
Wait for canceled or another terminal status, or complete an explicit reconciliation path that proves the old goal no longer has authority.
Job connection
How this becomes employable evidence
Implement a long-running mission executor or behavior tree that monitors ROS 2 action progress and state freshness, verifies effects, performs contextual versus system recovery within budgets, and exposes every replan and terminal decision to operations.
Relevant target roles
- Robotics Software Engineer — ROS 2 / AMR
- Robotics Application / ROS 2 Integration Engineer
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
Chapter 20 interview drill
Interview questions: Execute, monitor, re-observe, and replan long-horizon tasks
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A navigation action is still executing and publishing feedback, but the robot is not getting closer. Explain detection, cancellation, re-observation, contextual recovery, plan revision, retry budget, and the evidence required for recovered success.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why does the orchestrator need a verifying state after ROS action success?
The action result describes server execution, while verifying checks the fresh physical postcondition and intended entity before dependent skills continue.
Q2What distinguishes replanning from retrying?
Replanning uses updated authoritative state, records a new plan revision, and changes the invalid route or skill choice; retrying repeats an attempt, often with the same plan.
Q3When should recovery stop?
When its retry, time, or risk budget is exhausted, required state remains unreliable, no safe feasible plan exists, an invariant fails, or policy requires human authority.