Chapter 20 · Build a bounded language-to-ROS 2 task executor

Today in the field story

One problem, then the next

The planner says “pick up the blue card,” but two blue objects now exist. Courier-17 pauses at the grounding checkpoint, requests fresh candidates, and refuses to invent which entity the user meant. Once identity is resolved, it records pose, frame, confidence boundary, scene version, and age, then re-observes immediately before dispatch. The planted stale-entity starter captures the same principle in miniature. A noun becomes actionable only after it points to one current, authorized physical record.

Why now: Grounding prevents correct-looking plans from acting on the wrong physical object.
Ignore today: Ignore open-ended visual question answering and guessed referents.
Unlocks next: Fresh entity-bound skill requests with ambiguity as a terminally visible state.

Understand

Build the physical picture first

Grounding turns every planner noun into one timestamped entity record, then re-observation proves that record still describes the object about to be touched.

A phrase is not an actuator-safe reference. Ground “the red block” into a record containing stable entity ID, object class, relevant attributes, pose, coordinate frame, observation time, source, confidence, and scene version. Preserve the original phrase and candidate set for audit. If two candidates satisfy the language or the detector cannot support identity, the correct state is ambiguous, not whichever bounding box has the highest convenient score.

Separate detection, tracking, identity, and pose. A detector may find a red rectangle; a tracker may associate it across frames; an identity check may read a tag or use operator confirmation; pose estimation describes where the entity is in a named frame. Image overlap or confidence alone does not prove identity. Skills declare which evidence they need: coarse navigation may tolerate a zone, while picking may require current six-degree pose, reachable geometry, and an object-specific confidence threshold.

Plans age while people and robots move. Immediately before dispatch, reacquire the relevant entity or confirm a sufficiently fresh tracked observation, transform it into the skill's required frame, and compare scene version, pose delta, reservation, reachability, and workspace guards. If the entity moved beyond tolerance, invalidate only the affected plan suffix and return to grounding or planning. Never patch the goal pose silently while retaining evidence from the older target.

Observe again after execution to verify the intended effect and identity. A grasp command can succeed while the gripper is empty or holding the neighboring block; a place command can finish while the object is outside the target zone. Check held(entity_id) or in_zone(entity_id, zone_id) using fresh sensors, along with absence of unintended changes. This closes the loop between language, perception, skill, and physical outcome.

Words you need

Name each idea precisely

Grounding: Binding a language reference or symbolic plan argument to a specific perceived entity and its evidence.
Physical example:
“The red block beside the tape” becomes entity B-17 from snapshot 205, not a string forwarded to the gripper skill.
Entity ID: A stable identifier used to correlate observations, reservations, skill calls, and outcomes for one intended physical object.
Physical example:
B-17 remains the task target while its camera bounding box and base-frame pose change between observations.
Reference resolution: The process of generating candidates for a phrase, applying attributes and relations, and deciding resolved, ambiguous, missing, or invalid.
Physical example:
Two red blocks remain candidates until “nearest the yellow tape” or an operator selection identifies one.
Freshness window: The maximum observation age allowed for a particular skill decision before another observation is required.
Physical example:
A pick pose older than 300 ms is rejected even though a two-second-old pose may still be useful for choosing which camera to inspect.
Pose delta: The translation and rotation difference between the state used to plan and the most recent state used to validate execution.
Physical example:
The block moved 4 cm after planning, exceeding the pick contract's 1 cm tolerance and invalidating the grasp.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalTask graphs, timeouts, and retries

An autonomous task is a state machine with measurable guards, not one long prompt.

t_deadline: latest allowed completion timeUnit: seconds (s)
N_retry: maximum retry countUnit: attempts
P(success): observed success frequencyUnit: probability

A grasp skill gets a 5 s timeout and at most 2 retries.
The maximum planned attempt time is 3×5 = 15 s, excluding recovery.
After each failure, re-observe and check a typed precondition before retrying; do not replay a stale command.

Programmer analogy

It resembles a workflow engine with typed APIs, except retries require fresh perception of a changed world.

One initial try plus three retries, each capped at 4 s, permits how much attempt time?

4 attempts × 4 s = 16 s.

Possible motion during pose age is

d=vt=(0.25\ \mathrm{m/s})(0.4\ \mathrm{s})=0.10\ \mathrm{m}.

Since $0.10\ \mathrm{m}>0.02\ \mathrm{m}$ , the uncertainty exceeds the stated grasp tolerance.

Resolve, recheck, and pick the intended red block

A paper or simulated scene contains B-17 and B-22, both red cubes. B-17 is left of marker M1; B-22 is right of it. The user asks, “Put the red block next to the marker into bin C,” and the pick skill requires identity confidence, base-frame pose younger than 300 ms, and pose delta below 1 cm.

Parse the request into attributes color=red and relation next_to(marker M1) while retaining the original phrase. Query snapshot S31 and return both candidates with their evidence rather than immediately selecting one.
Apply the relation definition and find that both are within the broad 20 cm “next to” threshold. Mark the reference ambiguous and ask, “the block left or right of M1?” without moving the robot.
After the user chooses left, bind the goal to entity B-17 and store the candidate set, clarification, observation ID, pose, frame, confidence, and scene version in plan P4.
Before pick dispatch, acquire snapshot S32, reacquire B-17 by tag or validated track, transform its pose into robot_base, and compare age, confidence, reservation, reachability, workspace, and pose delta with the contract.
Inject a 4 cm movement between S31 and S32. Reject the P4 pick call as TARGET_MOVED, invalidate its approach and pick suffix, and preserve the unchanged bin and terminal-goal parts.
Generate P5 from S32, approach and pick B-17, then re-observe the gripper and scene. Require evidence that B-17 is held and B-22 remains at its prior location within tolerance.
Only after the held-identity postcondition passes may the plan call place in bin C; another fresh observation must then prove in_zone(B-17, C) and an empty gripper.

Result

The language reference resolves through explicit clarification, the moved target invalidates a stale plan before motion, and both grasp and placement close on the intended entity rather than controller status alone.

What this proves

Grounding is an evidence chain from phrase to candidate set to stable ID to fresh pose to verified effect; breaking any link returns to perception or clarification.

Physical examples

Where this appears in real life

Two red blocks near one bin

Blocks B-17 and B-22 are both red and inside the camera view; only B-17 is beside the yellow tape, but the user's request says only “pick the red block.”

Look for:

The resolver returns two candidates and requests clarification; it does not use detector confidence as a substitute for the user's missing distinction.

Object moved after planning

A grasp was planned from snapshot 31, then a person moves the foam block several centimetres before the executor reaches the pick step.

Look for:

Pre-dispatch observation produces a new scene version and out-of-tolerance pose delta, so the old goal is rejected and the affected plan suffix is rebuilt.

Hands-on exercise

Make the idea observable

Place two similar cards or foam blocks around one landmark, or reproduce the scene in simulation. Use only stationary tabletop manipulation unless a supervised commissioned platform is available.

Give every object and landmark a stable ID, then record attributes, pose, frame, timestamp, source, confidence, and scene version for one snapshot.
Write three referring expressions: one unique, one ambiguous, and one referring to a missing entity. Make the resolver return candidate evidence plus resolved, ambiguous, or missing state.
Define skill-specific freshness, confidence, pose-delta, workspace, and identity requirements; keep the planning threshold distinct from the stricter pick threshold.
Resolve the unique request, create a plan, then move the target before dispatch. Re-observe and prove the old call is rejected by scene version or pose delta.
Run the refreshed plan and deliberately simulate a false grasp in which the action returns success but the target remains on the table; block the place step on the failed held-object postcondition.
Repeat with a verified grasp and place, checking the intended entity, target zone, empty gripper, and absence of unintended movement before declaring success.

Observe

Language ambiguity, detector uncertainty, track identity, coordinate transforms, and state age fail in different ways; one confidence number cannot safely collapse them into a single “probably this object” decision.

Done when

Ambiguous and missing references never dispatch, moved targets invalidate stale goals, false controller success cannot satisfy the grasp, and the final trace identifies the exact entity placed.

Build today

Build an agent that converts a natural-language goal into inspectable ROS 2 actions and recovers from one failure.

Evidence to save

DONE when the integrated “Ground planner outputs in perceived entities and re-observe before action” path is observable, cancelable, and leaves the prior baseline reproducible.

Common mistakes

Catch the wrong mental model

Wrong

Passing the phrase “red block” or a raw bounding box directly to a manipulation skill.

Better

Resolve the phrase to a stable entity with identity evidence and a current frame-aware pose that satisfies the skill contract.

Wrong

Choosing the highest detector score when two objects satisfy the user's description.

Better

Return the candidate set and clarify the missing relation or identity; detector confidence measures a model output, not user intent.

Wrong

Reusing a planned pose after re-observation shows the entity moved.

Better

Reject the stale call, record the scene and pose delta, and rebuild the affected plan suffix from the new snapshot.

Job connection

How this becomes employable evidence

Connect a language planner to perception and ROS 2 skills through stable scene entities, frame-aware poses, freshness rules, ambiguity handling, and before-and-after observations that prevent the wrong package, tote, or tool from being manipulated.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 20 interview drill

Interview questions: Ground planner outputs in perceived entities and re-observe before action

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A user says “pick the red part,” two red detections exist, and one moves after planning. Walk through reference resolution, entity identity, frames, freshness, plan invalidation, and post-grasp verification.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What minimum evidence makes a planner's object reference actionable?

Model interview answer

A resolved stable entity ID with relevant attributes, current pose and frame, source and timestamp, sufficient confidence, scene version, and skill-specific preconditions.

Q2Why does high image overlap not prove object identity?

Model interview answer

Overlap shows spatial agreement between regions, while identity requires evidence that the region corresponds to the same intended physical entity across time and candidates.

Q3What should the executor do when pre-dispatch pose delta exceeds the skill tolerance?

Model interview answer

Reject the stale call, preserve the new observation, invalidate the affected plan suffix, and re-ground or replan before any motion.