Chapter 20 · Build a bounded language-to-ROS 2 task executor
Today in the field story
One problem, then the next
The Courier-17 review replays one delivery from the initial request to fresh terminal observation. A single correlation identity joins goal interpretation, grounded entities, plan versions, skill UUIDs, feedback, cancellation, postcondition checks, recovery, timings, and the final handoff or success. You preserve the stale-object rejection and the largest remaining failure beside the clean run. This execution trace becomes Week 21's attack surface: the next team will test whether semantic tricks, invalid actions, stale state, and exhausted recovery remain contained.
- Why now
A complete trace is the minimum evidence for debugging and adversarial evaluation.
- Ignore today
Do not edit away failed branches or call one run reliable.
- Unlocks next
A concrete planner-to-skill system ready for independent safety evaluation.
Understand
Build the physical picture first
A long-horizon demo is an evidence ledger: one run ID connects the original goal to every observation, plan revision, skill attempt, recovery, intervention, and final fact.
Assign a run ID before interpreting the user request. Every trace event then carries sequence number, monotonic and wall-clock time, run ID, task and plan revision, component, event type, request and goal IDs when present, input and output state versions, and a compact payload. Append events for clarification, grounding, plan acceptance, precondition rejection, dispatch, status, feedback, cancel, observation, effect verdict, retry, replan, compensation, handoff, and terminal outcome.
Preserve the distinction among proposed, accepted, commanded, measured, and verified state. A planner proposal is evidence of reasoning, not motion; action acceptance is evidence that a server took ownership, not completion; feedback describes progress; a result describes the action lifecycle; a postcondition observation supports the physical claim. Store plan revisions and their supersedes links so a reviewer can tell which steps actually ran and why an old suffix was abandoned.
Freeze the demo before running it. Declare initial scene, entities, reset, goal predicates, invariants, time and retry budgets, failure injector, allowed intervention, success, partial-success, failure, and unsafe-attempt rules. Run at least a nominal case and one changed-state case using the same versions. An uncut screen recording is useful for timing and human-visible behavior, but it complements rather than replaces structured state, action, and result records.
Report denominators and terminal categories. Measure verified task success, completion duration, planning and skill latency, retries, replans, recovery success, intervention count and duration, handoffs, stale or rejected calls, and unsafe attempts. A rough product of independent skill success rates can reveal chain fragility, but observed long-horizon evidence must retain conditional branches and shared causes. Publish the largest remaining failure and the exact next test rather than selecting only the clean run.
Words you need
Name each idea precisely
- Run ID
The stable identity joining all plans, calls, observations, logs, video markers, results, and reports for one end-to-end attempt.
Physical example:Run RUN-204 appears in the natural-language request event, ROS goal wrapper, simulator state capture, recovery record, and final report row.
- Trace event
An immutable timestamped record of one meaningful decision, state change, command lifecycle update, observation, or intervention.
Physical example:Event 38 records that goal G12 reached canceled and event 39 records zero commanded velocity before the new plan is accepted.
- Plan revision
A versioned plan tied to a specific input state and linked to the earlier plan it replaces or refines.
Physical example:P6 supersedes P5 after the target block moves; completed observation steps remain, while the approach and pick suffix changes.
- Intervention
A human action that changes task execution, state, or authority and therefore must be counted separately from autonomous recovery.
Physical example:An operator selects B-17 from two candidates; the clarification is logged and the run is not labelled fully autonomous.
- Verified terminal outcome
The final success, partial, failure, handoff, or abort category supported by fresh goal predicates, invariant checks, and robot safe-state evidence.
Physical example:The block is inside bin C and the gripper is empty, but the arm failed to park before deadline, so the declared outcome is partial rather than success.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalTask graphs, timeouts, and retries
An autonomous task is a state machine with measurable guards, not one long prompt.
- t_deadline
- latest allowed completion timeUnit: seconds (s)
- N_retry
- maximum retry countUnit: attempts
- P(success)
- observed success frequencyUnit: probability
A grasp skill gets a 5 s timeout and at most 2 retries.
The maximum planned attempt time is 3×5 = 15 s, excluding recovery.
After each failure, re-observe and check a typed precondition before retrying; do not replay a stale command.
It resembles a workflow engine with typed APIs, except retries require fresh perception of a changed world.
One initial try plus three retries, each capped at 4 s, permits how much attempt time?
4 attempts × 4 s = 16 s.
Under an independence approximation, five skill rates give
Use the correlated trace to measure actual duration, retries, branches, and final outcome.
Reconstruct a blocked-path storage run from its trace
Run RUN-204 asks a simulated mobile manipulator to move block B-17 from tray T1 to bin C. The frozen injector moves a trolley into the navigation path after the first goal starts; one replan is permitted.
Events 1–6 record request text, parsed terminal predicates and invariants, clarification-free grounding to B-17, initial snapshot S70, accepted plan P20, software versions, and the frozen injector configuration.
Events 7–15 record navigation request R31, ROS goal G31, acceptance, status, feedback, and current state. The trace shows feedback remained fresh while the progress predicate failed after the trolley appeared.
Events 16–22 record
NO_PROGRESS, cancellation request, canceling, canceled, zero commanded motion, snapshot S71 with trolley T8, and recovery-budget decrement. No new goal appears before G31 reaches a terminal state.Events 23–27 record plan P21 superseding P20, the changed navigation suffix, preserved B-17 and bin predicates, validated remaining deadline, and dispatch of new request R32 with goal G32.
Events 28–46 record alternate navigation, fresh grounding, pick, held-object verification, place, in-bin verification, empty gripper, parking, and invariant checks. Every succeeded action is followed by a separate state observation and postcondition verdict.
Compute the result from the ledger: one of one task succeeded, one of one injected failures recovered, one replan, zero human interventions, zero unsafe calls, and duration 74 seconds within the 90-second threshold. Keep skill and planning latencies as separate fields.
Rebuild the report and video markers using only RUN-204 events. If any decision cannot be explained or the final predicates cannot be recomputed from stored observations, mark the evidence incomplete and repair instrumentation before presenting the run.
RUN-204 supports recovered success because its obstacle, progress failure, cancellation, new state, plan revision, downstream skill effects, and terminal predicates are all correlated and recomputable.
An uncut demo becomes engineering evidence only when its visible behavior can be reconciled with versioned decisions, action lifecycle, fresh world state, and predeclared outcome rules.
Physical examples
Where this appears in real life
Correlated tabletop card run
A camera records a paper-block task while a second view shows a trace table whose events use the same run ID, object ID, plan revisions, and injected-failure marker.
A reviewer can pause at any physical change and find the proposal, validation, state snapshot, skill state, and later postcondition event that explains it.
Polished video with missing lineage
A robot video shows a parcel reaching a shelf, but the logs start after the goal, omit one restart, contain no plan version, and end at action success without a measured parcel identity.
The clip is a demonstration artifact but cannot prove the declared end-to-end outcome, autonomy level, recovery path, or reproducibility.
Hands-on exercise
Make the idea observable
Use the week's simulator or paper-world build. Freeze one nominal scenario and one scenario that changes an object, blocks a path, or causes a single skill postcondition to fail.
Write a trace schema with sequence, two clocks, run ID, task and plan revision, component, event type, request and goal IDs, state versions, payload, and evidence reference.
Declare both scenarios' resets, terminal predicates, invariants, time and recovery budgets, allowed intervention, failure injection point, and terminal outcome categories before execution.
Run the nominal case without edits, capturing request, grounding, plan, validations, skill status and feedback, observations, effect verdicts, timings, and final safe state under one run ID.
Reset to the frozen initial state and run the injected-failure case. Require a visible changed snapshot, bounded recovery decision, new plan revision when needed, and preserved record of the abandoned attempt.
Mark any operator action with identity, reason, duration, and authority change; do not label a clarified, rescued, or manually repositioned run fully autonomous.
Compute success denominator, durations, retries, replans, recovery result, interventions, rejected or stale calls, and unsafe attempts directly from trace events, then compare them with the predeclared thresholds.
Ask another person to reconstruct both runs and final predicates from the evidence bundle. Fix missing lineage, retain the largest failure, and record one next experiment tied to that failure category.
Missing IDs, state versions, or terminal observations create gaps exactly where a polished demo can hide duplicated commands, manual rescue, stale plans, or an incorrect object outcome.
Both frozen runs are reconstructable from one schema, every action result has an effect verdict, every replan names its input state and predecessor, interventions are counted, and the final category can be recomputed without trusting narration.
Build today
Build an agent that converts a natural-language goal into inspectable ROS 2 actions and recovers from one failure.
Evidence to save
DONE when the weekly ship note explains how “Long-horizon task demo with complete execution trace” changed the build, what still fails, and the first task for next week.
Common mistakes
Catch the wrong mental model
Using a success video as the only evidence for a long-horizon task.
Correlate the video with versioned goal, plan, call, status, observation, recovery, intervention, and final-predicate events under one run ID.
Overwriting the original plan and failed attempt after recovery succeeds.
Append the new snapshot and plan revision, link it to the predecessor, and retain the failed attempt and recovery-budget change.
Reporting average completion time without outcome denominator, interventions, or unsafe attempts.
Publish terminal counts, duration distribution, recovery and intervention measures, rejected calls, safety outcomes, scenario definitions, and every retained failure.
Job connection
How this becomes employable evidence
Instrument an embodied-agent mission so model proposals, ROS 2 goals, simulator or robot observations, retries, operator actions, and terminal business outcomes share lineage; then convert the trace into repeatable acceptance and incident evidence.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robot Fleet Backend / Platform Engineer
- Robot HMI / Control & Monitoring Engineer
Chapter 20 interview drill
Interview questions: Long-horizon task demo with complete execution trace
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
What would you log to prove a multi-step language-to-robot task recovered from one failure? Separate proposal, action lifecycle, physical verification, plan revision, intervention, and terminal outcome evidence.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Which identity should connect the user request, plan, ROS goals, observations, video, and final report?
One run ID, supplemented by stable request, goal, entity, snapshot, and plan-revision IDs for the events inside that run.
Q2Why must every succeeded action have a later effect-verdict event?
Action success reports the server lifecycle; a separate fresh observation determines whether the intended physical postcondition and entity identity actually hold.
Q3When is a recovered demo still not fully autonomous?
When a human clarified, repositioned, selected, resumed, or otherwise changed execution or authority; that intervention must remain counted even if the final task succeeds.
Chapter starter artifact
Gate a typed skill on fresh grounded state
Run one complete simulated Courier-17 delivery with a correlated trace, one deliberately stale-entity rejection, one bounded recovery or handoff, and a terminal result verified from fresh world state.
const nowMs = 10_000;
const skill = {
name: "place_card",
requiredEntityId: "card-blue-7",
requiredFrame: "table",
maxAgeMs: 250,
};
const freshEntity = { id: "card-blue-7", frame: "table", observedAtMs: 9_900 };
const plantedFailure = { id: "card-blue-7", frame: "table", observedAtMs: 9_500 };
const missingTimestamp = { id: "card-blue-7", frame: "table" };
const futureTimestamp = { id: "card-blue-7", frame: "table", observedAtMs: 10_100 };
function admit(entity) {
if (entity?.id !== skill.requiredEntityId) return "REJECT entity";
if (entity.frame !== skill.requiredFrame) return "REJECT frame";
if (!Number.isFinite(entity.observedAtMs)) return "REJECT timestamp";
const ageMs = nowMs - entity.observedAtMs;
if (ageMs < 0) return "REJECT future_entity";
if (ageMs > skill.maxAgeMs) return "REJECT stale_entity";
return "DISPATCH " + skill.name;
}
console.log("fresh observation: " + admit(freshEntity));
console.log("planted stale observation: " + admit(plantedFailure));
console.log("missing timestamp: " + admit(missingTimestamp));
console.log("future timestamp: " + admit(futureTimestamp));
console.log("mission state: HANDOFF");Download the file into your terminal's current folder, then run the command below. The expected output is exact.
- Run
node week-20-courier-17.mjs
- Expected output
fresh observation: DISPATCH place_card planted stale observation: REJECT stale_entity missing timestamp: REJECT timestamp future timestamp: REJECT future_entity mission state: HANDOFF
- Planted failure to diagnose
The stale card, absent timestamp, future timestamp, or wrong identity must each fail closed rather than dispatching with an invalid freshness calculation.