Chapter 15 · Turn demonstrations into measured, bounded robot policies

Today in the field story

One problem, then the next

Wrap the selected checkpoint in a clocked policy loop: reset state, acquire fresh input, reproduce processing, infer before expiry, decode named units, validate limits and rate, execute briefly, and observe again. Inject a 140 ms result like the starter’s c4 case and require rejection plus the declared hold, stop request, or handoff. A model answer is never allowed to become valid merely because it arrived eventually.

Why now: Evaluation needs the same timing and guard boundary that real integration would enforce.
Ignore today: Ignore safety-rated control and fleet deployment; prove one application-level fallback in simulation.
Unlocks next: A bounded policy runtime reusable by later embodied-AI systems.

Understand

Build the physical picture first

A policy runtime is a guarded clocked loop: acquire fresh evidence, predict before the deadline, decode and check one bounded action, execute briefly, then observe again or fail safely.

A deployable inference loop begins at episode reset, not at the model call. Reset clears action queues, temporal context, recurrent or policy state, timers, and previous-task identifiers. The loop then acquires a new observation with source timestamps and validates required cameras, robot state, task context, shape, finite values, and freshness. Preprocessing must match training exactly, including feature names, order, image crop, color order, normalization statistics, and missing-input policy. A tensor that enters the model successfully can still describe the wrong physical moment.

Inference has a deadline set by the control contract. At 30 Hz, one cycle lasts about 33.3 milliseconds, but sensing, transport, preprocessing, model work, postprocessing, guards, and command delivery all consume that budget. Measure end-to-end age and stage latencies with a monotonic clock; model latency alone omits waiting and stale inputs. Report median and tail values such as p95 or p99 plus deadline misses. A numerically good prediction arriving after its intended scene has changed is stale and must not be sent as if current.

Postprocessing converts model output back to the declared action representation. It must check output shape, dtype, finite values, component order, units, frame, normalization version, and supported mode. Then independent command guards apply robot and task constraints such as joint or velocity bounds, workspace zones, rate of change, collision or contact conditions, enable state, and command age. Blindly clipping a wildly invalid prediction to the nearest limit can hide a model or scaling defect; severe violations should be rejected and diagnosed rather than made to look normal.

Fallback is a predefined robot-system behavior, not the neural network's opinion. Depending on the commissioned system, it may hold position, command a controlled stop, stop sending motion and let a watchdog act, or request human control. The safe choice depends on gravity, brakes, contact, payload, traffic, and low-level controllers, so 'send zeros' is not universal. Log the reason, last accepted observation and action IDs, queue state, deadline, guard decision, and terminal outcome. A chunk queue also needs freshness limits so old actions cannot outlive the scene that justified them.

Words you need

Name each idea precisely

Control deadline: The latest permitted time for an observation-derived command to be validated and accepted for its intended control cycle.
Physical example:
At 30 Hz the nominal period is 33.3 milliseconds, so a command completing after the configured deadline is rejected rather than sent late.
Observation age: Elapsed time from the physical acquisition timestamp of an observation to the moment it is used for a command decision.
Physical example:
A camera frame captured 80 milliseconds ago may show a cup where it was, not where it is when the gripper command executes.
Postprocessing: The declared conversion from model outputs to physical action values, including de-normalization, representation, component order, and units.
Physical example:
Seven normalized outputs become seven target joint positions in radians using the training dataset statistics saved with the checkpoint.
Action guard: An independent check that accepts or rejects a decoded action according to runtime timing, validity, and robot or task constraints.
Physical example:
A guard rejects a non-finite joint target and a command that would cross the configured workspace boundary.
Safe fallback: A commissioned behavior entered when current policy output cannot be accepted, chosen from the physical system's hazards and controller design.
Physical example:
A mobile base performs a controlled stop after stale perception, while a gravity-loaded arm may require its existing safety controller to hold safely.
Action queue: An ordered buffer of predicted future commands with provenance, age, and replacement rules.
Physical example:
A chunked policy has six actions left, but the client discards them after a stop event or when their source observation exceeds its freshness limit.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalSupervised loss, rollout success, and the RL boundary

Low offline imitation error does not guarantee closed-loop recovery, while reinforcement learning adds a reward-driven decision process that can optimize unintended shortcuts.

MSE: average squared normalized action-component errorUnit: unitless
S/N: successful trials divided by all frozen trialsUnit: probability or percent
G = Σγᵗrₜ: discounted return used by an RL objectiveUnit: reward units

For one normalized, unitless action component, the errors are [1, −1, 2]. Square them: [1,1,4].
Average: MSE = (1+1+4)/3 = 2. Never average raw metres, radians, and discrete gripper values as though they share one unit.
Also run the policy: 16 successes from 20 frozen trials is 80%. If a later sim-only RL experiment is proposed, declare its state, actions, transitions, reward, discount, safety constraints, and reward-hacking tests before comparing the same suite.

Programmer analogy

A unit test can pass while the full user flow fails. RL additionally resembles optimizing a business KPI: a system may improve the number while violating the real intent.

What is success rate for 18 successes in 24 trials?

18/24 = 0.75 = 75%.

An independent-step reliability estimate is $0.99^{20}\approx0.818$ . Timing is separate:

r_{\mathrm{late}}=\frac{3}{100}=3\%.

A late action must be rejected by the deadline gate even when its predicted value looks correct.

Budget a 30 Hz inference cycle and reject one late command

A loop runs at 30 Hz, giving 1000 / 30 ≈ 33.3 ms per cycle. One nominal trace uses 8 ms for sensing and transport, 4 ms for preprocessing, 14 ms for model inference, 3 ms for postprocessing and guards, and 2 ms for command delivery. A second trace raises inference to 18 ms.

Calculate the nominal deadline period: 1000 ms / 30 ≈ 33.3 ms; record whether the actual configured deadline is equal to or shorter than this period.
Add nominal stages: 8 + 4 + 14 + 3 + 2 = 31 ms from acquisition path through command delivery.
Calculate nominal slack: 33.3 - 31 = 2.3 ms; that small margin must also tolerate scheduling variation and cannot be inferred from average model time alone.
Replace model time with the slow trace and add again: 8 + 4 + 18 + 3 + 2 = 35 ms, which misses the 33.3 ms period by about 1.7 ms.
Reject or invalidate the late action according to the runtime contract, enter the commissioned fallback, and log observation ID, age, each stage duration, guard result, and fallback result.
Repeat over many cycles to calculate miss rate and tail latency; one nominal 31 ms sample is not proof that the loop meets its deadline reliably.

Result

The nominal trace uses 31 ms with only 2.3 ms slack; the slow trace takes 35 ms and must not execute as an on-time command.

What this proves

Runtime fitness is measured from fresh observation to accepted physical command, including tail behavior and fallback—not by model speed in isolation.

Physical examples

Where this appears in real life

Out-of-date direction card

A partner shows a left-arrow card, then moves a paper obstacle before you finish a slow calculation. Compare following the old arrow with checking the card's timestamp and stopping.

Look for:

Correctly reading a stale arrow is still the wrong current action; value accuracy and timing validity are separate gates.

Conveyor queue after a stop sign

Place six action cards on a paper conveyor. Insert a red stop event after the second card and decide whether the four queued cards may continue.

Look for:

A queue needs invalidation and provenance. Actions that were reasonable before the stop cannot execute simply because they were already predicted.

Hands-on exercise

Make the idea observable

Build a local mock loop using synthetic timestamped observations and actions, or use a bounded simulator with actuation disabled. Include a visible event log.

Define the cycle frequency, observation freshness limit, command deadline, required fields, action units and bounds, queue invalidation events, and task-specific fallback before running.
Implement or lay out reset, acquire, validate, preprocess, infer, postprocess, guard, execute-or-fallback, and log as separately timed stages.
Feed one nominal observation, one missing feature, one NaN output, one wrong-shaped chunk, one old timestamp, and one deliberately slow inference result.
Verify invalid cases never reach the execution sink and each produces a distinct reason plus the expected fallback event.
Run at least 100 synthetic cycles with varied inference delays; report median, p95, maximum, deadline-miss count, and any empty or stale queue event.
Save the configuration, event ledger, one accepted trace, every rejection type, and a statement of what the disconnected mock cannot prove about a real robot.

Observe

Model outputs can be numerically plausible yet stale, malformed, wrongly scaled, outside limits, or left in a queue after their source situation is invalid.

Done when

Every stage and clock boundary is inspectable, all injected faults fail closed at the intended gate, totals reconcile across 100 cycles, and no physical safety claim is made from the mock.

Build today

Train behavior-cloning and ACT-style policies on the same task and compare rollout behavior.

Evidence to save

DONE when a 60–120 second uncut “Policy inference loop, latency, and safe fallback” demo links to its command, logs or plots, result count, and honest failure note.

Common mistakes

Catch the wrong mental model

Wrong

Measuring only the model call and declaring the control deadline met.

Better

Measure acquisition age, waiting, preprocessing, inference, decoding, guards, transport, and command acceptance with tail latency and miss counts.

Wrong

Clipping every invalid output into the legal numeric range.

Better

Reject non-finite, wrongly scaled, malformed, stale, or severely out-of-range proposals because clipping can conceal a broken policy boundary.

Wrong

Using 'send zero' as a universal safe state.

Better

Choose and commission fallback from the robot's mechanics, low-level controller, brakes, gravity, payload, contact, and workspace hazards.

Wrong

Resetting the environment while leaving policy context or action queues intact.

Better

Clear every episode-scoped buffer and policy state, associate new observations with the new task identity, and reject actions from the previous episode.

Job connection

How this becomes employable evidence

Integrate a learned policy behind feature, freshness, deadline, de-normalization, action-limit, queue, watchdog, cancellation, and fallback boundaries, then expose the decisive timing and rejection state to operators and validation logs.

Relevant target roles

Robotics Software Engineer — ROS 2 / AMR
Robotics Deployment, Integration & Validation Engineer
Robot HMI / Control & Monitoring Engineer
Robot Learning Deployment / Physical AI Integration Engineer

Chapter 15 interview drill

Interview questions: Policy inference loop, latency, and safe fallback

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A policy's model inference is 20 ms on a 30 Hz controller, yet commands still arrive late. Build the complete timing budget, show how you detect stale observations and queued actions, and describe a system-specific fallback.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why can a correct predicted value still be rejected?

Model interview answer

It may be based on an old observation or arrive after its deadline, so it no longer describes a valid command for the current physical scene.

Q2What belongs in the end-to-end latency budget?

Model interview answer

Observation acquisition and age, transport or waiting, preprocessing, inference, postprocessing, guards, command delivery, and acceptance—not only the model call.

Q3Why is fallback specific to the robot system?

Model interview answer

Holding, stopping, sending no command, or commanding zero can have different consequences depending on gravity, brakes, contact, payload, traffic, and controller behavior.