Chapter 20 · Build a bounded language-to-ROS 2 task executor
Today in the field story
One problem, then the next
Courier-17 remembers yesterday's successful card location, but today's camera reports an empty mark. You split current world state from append-only episode history, attach version and timestamp to observations, and retrieve memories only as hypotheses or recovery clues. Conflicts resolve toward fresh trusted perception, not the most convenient prior answer. The mission ledger shows exactly which record influenced a plan, preventing a stale embedding result from masquerading as present physical truth when the card or tray has moved.
- Why now
Long-horizon plans need memory without allowing memory to overwrite reality.
- Ignore today
Ignore general vector databases and unbounded conversational memory.
- Unlocks next
Versioned current state plus auditable, scoped historical evidence.
Understand
Build the physical picture first
World state is a timestamped whiteboard of what is trusted now; episodic memory is a journal of what happened before, never a substitute for looking.
Represent current world state as versioned facts about specific entities, not as one prose summary. An entity record can contain stable ID, type, pose, frame, attributes, observation time, source sensor or service, confidence, validity interval, and reservation or ownership. Robot records include mode, active goal, localization quality, tool state, battery, stop state, and authority. A writer creates a new snapshot or atomic update; planners read one declared version so their preconditions do not mix facts from different moments.
Episodic memory records completed events and their context: task ID, plan revision, relevant observations, skill call, outcome, recovery, intervention, and lesson or label. Append history rather than rewriting a failed attempt into a success story. Store source and schema version with each event. Memory can answer “which recovery worked for this docking error under similar conditions,” but it must not assert that yesterday's charger, cup, person, or obstacle remains where it was.
Retrieval selects candidate evidence; it does not certify truth. Filter by robot or embodiment, site, skill and contract version, object class, failure code, operating conditions, recency, and permissions before ranking semantic similarity. Return event IDs and decisive fields so the orchestrator can inspect why an item matched. Measure relevance within the retrieved set, but also measure staleness, wrong-scope retrieval, and whether using the memory improved a frozen trial without unsafe attempts.
Define conflict and expiry rules before planning. Fresh trusted perception within the same scope normally overrides an older remembered location. A memory may seed where to look or which bounded recovery to consider, then the executor rechecks every current precondition. If two fresh sources disagree, mark the fact unknown, preserve both observations, and trigger another sensor, clarification, or handoff instead of choosing the more convenient story.
Words you need
Name each idea precisely
- World-state snapshot
An immutable versioned view of current robot, entity, environment, and authority facts used together for one planning or validation decision.
Physical example:Snapshot 84 says block B-17 is on tray T1 at a stamped base-frame pose, the gripper is empty, and the arm is parked.
- Scene version
A monotonically changing identity that lets a caller detect that relevant perceived state changed after its plan or request was formed.
Physical example:A person moves the block, producing scene 85; a pick call built from scene 84 is rejected and must re-observe.
- Episodic memory
An append-only record of a particular past task, observation, action, outcome, recovery, or intervention with provenance.
Physical example:Episode E-442 records that docking failed on a wet marker, a second camera view resolved the marker, and the retry succeeded once.
- Provenance
Metadata identifying where a fact came from, when it was produced, under which schema and configuration, and how it may be trusted.
Physical example:A cup pose includes camera C2, calibration version 7, timestamp, detector version, and confidence rather than only x-y-z values.
- Retrieval scope
The explicit boundary of robot, site, skill version, time, permissions, and conditions inside which a past event may be considered relevant.
Physical example:A recovery from gripper G1 firmware 3 is excluded when the current robot uses a different gripper and failure code.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalTask graphs, timeouts, and retries
An autonomous task is a state machine with measurable guards, not one long prompt.
- t_deadline
- latest allowed completion timeUnit: seconds (s)
- N_retry
- maximum retry countUnit: attempts
- P(success)
- observed success frequencyUnit: probability
A grasp skill gets a 5 s timeout and at most 2 retries.
The maximum planned attempt time is 3×5 = 15 s, excluding recovery.
After each failure, re-observe and check a typed precondition before retrying; do not replay a stale command.
It resembles a workflow engine with typed APIs, except retries require fresh perception of a changed world.
One initial try plus three retries, each capped at 4 s, permits how much attempt time?
4 attempts × 4 s = 16 s.
Precision among the top three retrieved memories is
This measures relevance within three results; it does not measure freshness, truth, or downstream plan success.
Resolve a stale memory before planning a mug delivery
The user asks for mug M-4. Current snapshot S120 contains a clearly observed empty shelf S2, mug M-7 on a cart, and no current record for M-4. Episodic search returns E88 saying M-4 was placed on S2 yesterday and E91 saying similar mugs are sometimes moved to wash station W1.
Write the current query as
locate entity_id M-4 at site lab-A, and preserve snapshot S120 rather than converting the request intopick from S2.Filter memories by site, entity or class, observation source, schema, permission, and time. E88 is entity-specific but old; E91 is class-level background and cannot establish M-4's location.
Compare sources: the current trusted view explicitly observes S2 empty, so it invalidates the remembered current-location claim from E88 while retaining E88 as a historical event.
Update world state to
location(M-4)=unknown, attach the S2 negative observation and conflicting episode IDs, and increment the scene version. Do not copy W1 into the current location field.Use scoped history only to order information-gathering steps: observe W1, query the inventory service, then ask the operator if still unresolved. Each step has a deadline and no motion begins from an unverified location.
Suppose W1 perception finds a mug but its label is occluded. Store the candidate and confidence, then request another view or operator confirmation instead of grounding M-4 by appearance alone.
Append the locate attempt, conflict resolution, observations, and final outcome as a new episode so later retrieval can distinguish a confirmed location, an unsuccessful search, and a human-provided identity.
The world model remains honestly unknown until fresh evidence identifies M-4, old memories guide a bounded search without overriding perception, and the conflict becomes traceable training and evaluation data.
Memory is useful when it narrows where to inspect or which recovery to test; it becomes dangerous when a retrieved sentence is promoted directly into current physical state.
Physical examples
Where this appears in real life
Yesterday's shelf location
A memory says mug M-4 was on shelf S2 yesterday, while the current camera snapshot sees S2 clearly and does not contain M-4.
The old episode suggests S2 as a search location but cannot create a current at(M-4, S2) fact; the plan observes another area or asks a person.
Dock recovery across robot versions
The agent retrieves three prior MARKER_LOST episodes, but two belong to a different camera mount and one matches the current robot, lighting range, skill version, and failure code.
Scope filters remove incompatible events before similarity ranking, and the matching recovery remains a candidate that current guards must validate.
Hands-on exercise
Make the idea observable
Use paper entity cards or a simulator snapshot plus a small JSON-like event ledger. Include two objects that can change location and at least one stale past event.
Define a world-state record with snapshot ID, entity ID, type, pose or unknown location, frame, source, observation time, confidence, validity, and reservation fields.
Create three successive snapshots by moving one paper object and changing one robot fact; verify each planner read names exactly one snapshot version.
Create five episodic records with task, plan, observation, skill version, outcome, error, recovery, intervention, and source fields. Preserve one failure rather than editing it into the later success.
Write a retrieval query for one failure or object-location question, then filter by site, robot or embodiment, skill version, failure code, recency, and permission before ranking relevance.
Inject a conflict between a recent observation and an older memory. Apply the written expiry and conflict rule, mark uncertainty explicitly, and choose re-observation, clarification, or handoff.
Score the top three results for relevance and wrong scope, then document which fields influenced the next bounded information-gathering step and which fields were forbidden from becoming current facts.
A high semantic match can still be obsolete, cross-robot, or based on an earlier schema; provenance and scope filters often remove the most linguistically similar but operationally wrong memory.
No stale episode overwrites current state, every fact used for a precondition has source and time, conflicting evidence produces an unknown or re-observe state, and the new attempt is appended with its real outcome.
Build today
Build an agent that converts a natural-language goal into inspectable ROS 2 actions and recovers from one failure.
Evidence to save
DONE when a deterministic “World state, episodic memory, and retrieval” failure test reports expected versus actual behavior and passes after the documented fix.
Common mistakes
Catch the wrong mental model
Using one vector database entry as both historical memory and authoritative current state.
Maintain a versioned current-state store with freshness and provenance; let retrieval return historical candidates that must be reconciled with that store.
Ranking by semantic similarity before filtering robot, site, skill version, time, and permission.
Apply hard operational scope first, then rank the remaining records and expose the fields that justified each match.
Overwriting a failed episode after a later retry succeeds.
Append the failure, recovery decision, and retry as distinct correlated events so evaluation and incident review retain the true sequence.
Job connection
How this becomes employable evidence
Build a scene and episode service that versions current robot facts, preserves task history, retrieves recovery evidence across a fleet without crossing robot or site scope, and makes stale or contradictory state visible before an agent dispatches work.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robot Fleet Backend / Platform Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
Chapter 20 interview drill
Interview questions: World state, episodic memory, and retrieval
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Design world state and episodic memory for a service robot. Explain entity identity, snapshot consistency, provenance, retrieval scope, conflict resolution, and why a remembered location cannot satisfy a current pick precondition.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What is the operational difference between world state and episodic memory?
World state is the versioned authoritative view used for current decisions; episodic memory is past evidence that may guide search or recovery but cannot satisfy a current physical precondition by itself.
Q2What should happen when a fresh camera view contradicts yesterday's remembered object location?
Keep the historical event, update current location to the fresh verified value or unknown, log the conflict, and re-observe or clarify before physical action.
Q3Why is precision among retrieved memories insufficient?
A relevant result may still be stale, wrong-scope, incompatible with the current robot or skill, or false; provenance, freshness, scope, and downstream trial effects must also be checked.