Chapter 18 · Trace vision and language into bounded robot action

Today in the field story

One problem, then the next

Resolve “the red mug beside the plate” against fresh entity IDs, attributes, relations, frames, timestamps, and confidence. With two plausible mugs, clarification is the correct system state. After the operator specifies mug-2, re-observe before acting because the scene may have changed. Preserve original text, candidates, clarification, chosen entity, preconditions, and rejection reason so language diversity does not erase physical grounding.

Why now: An action representation cannot be used until language refers to one current physical entity and admissible goal.
Ignore today: Ignore open-ended dialogue quality; prove one bounded grounding and clarification protocol.
Unlocks next: A typed, fresh task binding eligible for policy evaluation.

Understand

Build the physical picture first

Grounding is a courtroom identification: every noun, relation, and requested effect must point to visible evidence before a robot skill is admissible.

Language grounding connects words to the current physical scene and robot state. The instruction pick the mug beside the plate should resolve to a stable perceived entity ID, a relation evaluated in a named frame, and a task predicate—not merely to pixels that resemble training examples. Object labels alone are insufficient when there are duplicates. A grounded record can include candidate IDs, attributes, 2D or 3D location, relation scores, uncertainty, observation time, and the exact evidence used to select one referent.

The verb and goal also need grounding. Pick should map to an admissible skill with typed arguments, preconditions, effects, limits, and failure codes. A detected mug may be outside reach, already held, occluded, too hot, or not permitted to move. Affordance is task- and embodiment-specific: a handle may support a two-finger grasp for one gripper but not another. The planner or VLA does not get to convert recognition confidence into execution authority; the skill boundary checks current world and robot conditions.

Instruction diversity tests whether the system handles different valid ways of expressing the same intent. Paraphrases such as lift the crimson cup and pick up the red mug can share an expected referent and effect in a carefully constructed scene. Changing left of to right of, adding not, or changing cup to bowl is not a paraphrase; it changes the expected behavior. Build prompt groups with explicit semantic equivalence, dialect and word-order variation, relational language, corrections, and negative cases instead of counting every altered string as useful diversity.

Ambiguity is an expected system state. If two red mugs are equally plausible, the correct outcome is a clarification question or a safe rejection, not a confident guess. The HMI should present the interpreted entity, goal, and uncertainty before consequential execution when the workflow allows review. After a person clarifies, re-ground against a fresh scene because objects may have moved. Store the original instruction, interpretation, clarification, entity version, action decision, and outcome so failures can be attributed to perception, grounding, planning, policy, or execution.

Words you need

Name each idea precisely

Referring expression: Words used to identify an entity through category, attributes, location, or relation to other entities.
Physical example:
The small red mug beside the blue plate narrows several visible containers to one candidate using attributes and a spatial relation.
Grounding: Linking language to specific perceived entities, relations, states, skills, and expected physical effects in the current world.
Physical example:
The token sequence that cup is resolved to tracked entity cup-17 at a timestamped table pose before a pick skill is considered.
Affordance: An action opportunity supported by the entity, robot body, current state, and task constraints.
Physical example:
A wide handle affords a parallel-gripper grasp only while it is visible, reachable, cool enough, and not blocked by the plate.
Paraphrase: A different wording that preserves the same intended referent, relation, goal, and acceptance condition for the tested scene.
Physical example:
Put the red cube in the tray and place the crimson block inside the tray can share one expected result when labels and scene are controlled.
Ambiguity: A state in which available evidence supports multiple materially different interpretations without a declared winner.
Physical example:
Two red mugs are equally near the plate, so pick the red mug has two feasible referents.
Clarification policy: A rule specifying when to ask a targeted question, safely reject, or require operator selection instead of guessing.
Physical example:
When the best two referent scores are within the configured margin, the HMI asks, left mug or right mug? and commands nothing.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalToken, action, and latency budgets

Language reasoning must finish before the physical situation becomes stale.

T_total: end-to-end decision latencyUnit: milliseconds (ms)
f_control: safety/control update rateUnit: hertz (Hz)
H: action horizonUnit: steps or seconds

Perception takes 80 ms, model reasoning 220 ms, and skill dispatch 20 ms.
T_total = 80+220+20 = 320 ms.
At 1 m/s the robot moves 0.32 m during that delay; independent fast safety control must not wait for the LLM.

Programmer analogy

An LLM tool call resembles backend orchestration, but its latency must be converted into physical travel distance.

At 0.5 m/s, how far does a robot move during 400 ms?

400 ms = 0.4 s; distance = 0.5×0.4 = 0.2 m.

Measure paraphrase coverage as

C=\frac{N_{\mathrm{valid\ phrasings\ tested}}}{N_{\mathrm{target\ phrasings}}}=\frac{8}{10}=80\%.

This checks language-test coverage, not whether a phrase was grounded to the correct physical object.

Resolve one relational instruction and one ambiguous instruction

A paper scene contains M1 red mug at (0.20, 0.10) m, M2 red mug at (0.62, 0.12) m, P1 blue plate at (0.66, 0.10) m, and B1 green bowl at (0.40, 0.45) m, all in table frame at time 8.000 s.

Parse pick the red mug beside the blue plate into target category mug, attribute red, relation beside, reference category plate, reference attribute blue, and requested skill pick.
Filter entities by category and attributes: target candidates are M1 and M2, while the unique reference candidate is P1.
Define beside for this exercise as planar centre distance at most 0.10 m; calculate M1-to-P1 distance about 0.46 m and M2-to-P1 distance about 0.045 m.
Resolve M2 as the unique relational match, then verify its pose age, table frame, tracking confidence, reachability, gripper compatibility, workspace, and the preconditions registered by pick.
For pick the red mug, retain both M1 and M2; because category and color do not select one, return needs_clarification and ask a question using stable attributes such as left mug or mug beside the plate?.
After the answer, obtain a fresh scene snapshot, resolve the clarified expression again, and log original text, candidates, relation calculation, clarification, chosen entity version, skill decision, and later measured outcome.

Result

The relational instruction resolves to M2 under an explicit 0.10 m rule, while the shorter instruction causes no action because two referents remain plausible.

What this proves

Grounding is an auditable join between language and world state; uniqueness and admissibility are decisions, not side effects of fluent text.

Physical examples

Where this appears in real life

Two red mugs and one blue plate

A table holds red mugs M1 and M2. Only M2 is beside blue plate P1, while both mugs satisfy the category and color words.

Look for:

The relational phrase selects M2 only if beside has a declared geometric rule, the entities are fresh, and the result is unique enough for the clarification policy.

Correct object, inadmissible pick

The system correctly identifies a glass cup, but it sits outside the arm workspace behind a transparent barrier.

Look for:

Grounding can succeed while the pick affordance and skill preconditions fail; the system reports unreachable or blocked instead of turning perception confidence into motion.

Hands-on exercise

Make the idea observable

Use four labelled paper objects on a measured grid or a JSON scene graph. The exercise ends at a mock skill validator or simulator.

Create entity records with stable ID, category, attributes, table-frame pose, visibility, confidence, timestamp, reachability, and allowed manipulation state.
Write twelve instructions: four true paraphrases, three relation changes, one negation, two ambiguous references, one unknown object, and one request whose object is identified but unreachable.
For each instruction, define the expected entity ID or no-action state, goal predicate, and whether the correct route is execute in simulation, clarify, or reject.
Implement simple exact-match filters and measured left, right, inside, and near relations; return all candidates and evidence instead of only the top candidate.
Run the set twice, moving one object between runs; require fresh timestamps and verify a prior clarification cannot silently authorize an action on the moved scene.
Report referent accuracy, clarification correctness, false-execution count, and skill-precondition rejections separately; retain every failed case with its candidate set.

Observe

Small prepositions and negations change expected action, valid paraphrases preserve it, and correct object recognition can still end in a necessary skill rejection.

Done when

Every instruction has a predeclared expected outcome, ambiguous and unknown cases command nothing, moved objects require re-grounding, and the report exposes rather than averages away false executions.

Build today

Build a tiny language-conditioned policy interface over your existing simulator or dataset.

Evidence to save

DONE when a comparison table for “Language grounding and instruction diversity” contains the test condition, metric, result, and justified engineering decision.

Common mistakes

Catch the wrong mental model

Wrong

Using shared words as proof that an instruction refers to the correct object.

Better

Resolve all candidate entity IDs with attributes, relations, frames, timestamps, and uncertainty, then require a unique acceptable interpretation or clarification.

Wrong

Counting any changed sentence as instruction diversity.

Better

Separate semantics-preserving paraphrases from relation, object, negation, and goal changes, and assign an explicit expected outcome to each.

Wrong

Letting the model guess when two referents are plausible.

Better

Return a no-action ambiguity state, ask a targeted question or require operator selection, then re-ground the answer against a fresh scene.

Job connection

How this becomes employable evidence

Build the language-to-skill boundary that exposes perceived candidates, grounded entity IDs, relation evidence, ambiguity, operator clarification, precondition failures, and replayable outcome records instead of hiding interpretation inside a model response.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robot HMI / Control & Monitoring Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Deployment, Integration & Validation Engineer

Chapter 18 interview drill

Interview questions: Language grounding and instruction diversity

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Design grounding for pick the red mug near the plate when two red mugs exist. Define the scene representation, relation rule, ambiguity threshold, clarification UX, skill preconditions, and tests for negation, moved objects, and false execution.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why does `pick the red mug beside the blue plate` select M2 in the worked scene?

Model interview answer

M2 is the only red mug within the declared 0.10 m beside distance of the unique blue plate; M1 is about 0.46 m away.

Q2Can correct referent grounding make a pick automatically admissible?

Model interview answer

No. Reachability, visibility, gripper compatibility, workspace, object state, authorization, and every skill precondition still need independent validation.

Q3What should happen after a user clarifies an ambiguous object?

Model interview answer

Capture fresh world state and ground the clarified expression again before any skill request, because the scene or tracked entity may have changed.