Chapter 15 · Turn demonstrations into measured, bounded robot policies
Today in the field story
One problem, then the next
The handover depends on recent motion, so compare a single observation, observation history, and predicted action chunk without confusing them. Near the episode end, mask padded targets rather than training zeros as teacher behavior. Execute only a declared chunk prefix, re-observe, and measure how long an obsolete action can remain queued. The crooked approach should trigger a timely update instead of replaying an attractive old sequence.
- Why now
Temporal context can improve coherence while increasing stale-action exposure.
- Ignore today
Ignore long-horizon planning; reason about bounded history, prediction, and execution horizons.
- Unlocks next
A responsive chunk contract suitable for an ACT-style policy.
Understand
Build the physical picture first
Temporal context looks backward to understand what is happening, while an action chunk looks forward by proposing several ordered commands from one policy query.
One observation can be ambiguous. A single image of a gripper beside a cup does not reveal whether the gripper is approaching, pausing, or moving away. Temporal context supplies recent observations or states so the policy can estimate motion, task stage, and events hidden in one frame. A context window shaped [B, C, ...] contains C past or current steps for each batch example. Those steps need declared spacing, timestamp order, padding, and episode boundaries; history from the previous episode must never leak across reset.
An action chunk is an ordered sequence of future action targets predicted together. With batch size B, chunk horizon H, and action dimension A, the usual output contract is [B, H, A]. A ten-step chunk for a seven-value action contains 70 values per example. Predicting them jointly can preserve a coherent short strategy and reduce how many independent high-level choices a long episode needs. It does not make the low-level controller optional, and it does not guarantee those future commands remain appropriate after the world changes.
Prediction horizon and execution horizon are different. A policy may predict 20 future actions yet execute only the first four before observing and replanning. Executing all 20 is more open-loop and can continue toward an outdated target. Replanning every step is reactive but costs more inference and can produce inconsistent overlapping proposals. Some systems combine predictions for the same future timestep from overlapping chunks, such as ACT's temporal ensemble. Document which prediction ages are combined, their weights, and how unavailable or stale chunks are rejected.
Training needs chunk targets that stop cleanly at episode boundaries. A sample near an episode end may have fewer than H valid future actions, so it needs a validity or padding mask; padded zeros are storage, not teacher behavior, and must not contribute as genuine targets. Compare horizons with frozen rollouts, not only training loss. Measure success, reaction to a changed target, command smoothness, inference time, queue starvation, and how long an obsolete action can still execute.
Words you need
Name each idea precisely
- Temporal context
An ordered window of recent observations, states, or actions supplied so a policy can infer motion and task stage.
Physical example:Three camera frames show that a ball is rolling left even though the newest frame alone shows only its current position.
- Action chunk
A policy output containing several future actions in time order rather than only the next action.
Physical example:One query proposes four joint targets that move toward, close on, lift, and hold a paper gripper.
- Prediction horizon
The number of future action steps produced by one policy prediction.
Physical example:A horizon of 12 at 20 Hz covers 0.6 seconds of proposed commands.
- Execution horizon
The number of actions consumed from a predicted chunk before a fresh observation causes replanning or replacement.
Physical example:The policy predicts 12 steps but executes three, so it normally re-observes after 0.15 seconds at 20 Hz.
- Padding mask
A Boolean marker distinguishing valid future-action targets from placeholder positions beyond an episode's end.
Physical example:If only three actions remain in a five-step target, the mask
[valid, valid, valid, pad, pad]prevents two zeros from being learned as commands.- Temporal ensemble
A rule that combines overlapping predictions aimed at the same execution time, often weighting them by prediction age.
Physical example:Three earlier chunks each proposed the action for time 18; their declared weighted combination supplies one smoother command for that time.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalSupervised loss, rollout success, and the RL boundary
Low offline imitation error does not guarantee closed-loop recovery, while reinforcement learning adds a reward-driven decision process that can optimize unintended shortcuts.
- MSE
- average squared normalized action-component errorUnit: unitless
- S/N
- successful trials divided by all frozen trialsUnit: probability or percent
- G = Σγᵗrₜ
- discounted return used by an RL objectiveUnit: reward units
For one normalized, unitless action component, the errors are [1, −1, 2]. Square them: [1,1,4].
Average: MSE = (1+1+4)/3 = 2. Never average raw metres, radians, and discrete gripper values as though they share one unit.
Also run the policy: 16 successes from 20 frozen trials is 80%. If a later sim-only RL experiment is proposed, declare its state, actions, transitions, reward, discount, safety constraints, and reward-hacking tests before comparing the same suite.
A unit test can pass while the full user flow fails. RL additionally resembles optimizing a business KPI: a system may improve the number while violating the real intent.
What is success rate for 18 successes in 24 trials?
18/24 = 0.75 = 75%.
Under the simplifying independence assumption,
Fewer chunk-level decisions can reduce compounding opportunities, but longer chunks may react more slowly; test horizon and responsiveness directly.
Check chunk shape, coverage, padding, and reaction delay
A policy receives four recent state steps, predicts action chunks with H = 8 and A = 6, runs at 20 Hz, and executes E = 3 actions before replanning. A training sample has only five valid future actions before terminal.
Separate the contracts: the four-step input context looks backward, while the eight-step output horizon looks forward; neither number implies the other.
Calculate output size per example:
H × A = 8 × 6 = 48action values, with ordered axes[future_step, action_component].Calculate predicted time coverage:
8 / 20 = 0.40seconds of future actions.Calculate nominal re-observation interval:
E / 20 = 3 / 20 = 0.15seconds; a new event just after replanning may influence execution only after roughly that interval plus sensing and inference latency.Build the target validity mask as five valid steps followed by three padded steps; exclude the final
3 × 6 = 18placeholder values from reconstruction loss.Compare an execution horizon of three with executing all eight: the latter can commit for 0.40 seconds, so test whether its smoother motion is worth the longer stale-command exposure.
Each prediction contains 48 values covering 0.40 seconds; normal replanning occurs after 0.15 seconds, and 18 padded target values must not be treated as teacher actions.
Chunk design is a timing contract among target construction, tensor shape, replanning, masking, responsiveness, and low-level execution.
Physical examples
Where this appears in real life
Moving coin versus committed pointer
Move a cardboard pointer toward a coin in four marked steps while a partner shifts the coin after step one. Compare executing the remaining three old arrows with looking again and replacing them.
A long execution horizon follows a stale plan, while shorter execution reacts sooner but requires a stable rule for changing from one chunk to the next.
Which way is the toy car going?
Show one photo of a toy car at the centre of a ruler, then show three ordered photos with positions 8, 6, and 4 centimetres.
The single position is ambiguous, while ordered context reveals direction and approximate speed; context and future action output solve different problems.
Hands-on exercise
Make the idea observable
Use paper arrows, a coin target, a stopwatch, and a table with columns for observation time, context IDs, predicted chunk, executed prefix, and replacement reason.
Create two visually identical current states whose preceding three cards show opposite movement; explain why current-frame-only input cannot distinguish them.
Write action chunks of lengths one, four, and eight for moving a pointer toward the coin, keeping action units and time step constant.
For each chunk length, declare how many actions will execute before re-observation and calculate prediction coverage plus nominal replacement interval.
Ask a partner to shift the target on a hidden chosen step, then record how many outdated arrows each execution rule consumes before it reacts.
Create one near-terminal sample, pad it to the full chunk horizon, and show which target positions the loss mask excludes.
Choose one horizon and replacement rule using the measured reaction, smoothness, and workload table; retain the failed or slow trials with the decision.
Longer committed prefixes need fewer decisions but continue stale intent longer, while frequent replacement reacts sooner and can expose disagreement between overlapping chunks.
The table distinguishes context, prediction horizon, execution horizon, mask, and replacement timing, and the chosen rule follows from recorded trials rather than preference.
Build today
Train behavior-cloning and ACT-style policies on the same task and compare rollout behavior.
Evidence to save
DONE when a deterministic “Temporal context and action chunking” failure test reports expected versus actual behavior and passes after the documented fix.
Common mistakes
Catch the wrong mental model
Using temporal context and action chunk as two names for the same sequence.
Context is past or present input used to understand state; a chunk is future output proposed for execution, and each has its own length and mask.
Assuming every predicted action must execute before the policy can observe again.
Declare a separate execution horizon; a system can predict a longer chunk, consume a bounded prefix, and replan from a newer observation.
Filling missing terminal targets with zeros and including them in loss.
Mark targets beyond the episode boundary as padding and exclude them, because zero placeholders are not demonstrated stop commands.
Calling a longer chunk safer because it looks smoother in one video.
Measure stale-command duration, response to changes, limits, queue behavior, and closed-loop outcomes; visible smoothness alone does not establish safety.
Job connection
How this becomes employable evidence
Define the chunk tensor and padding contract between dataset and policy, implement reset-safe context buffers and bounded action queues, then benchmark horizon and replanning choices against reaction, smoothness, latency, and task success.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robotics Deployment, Integration & Validation Engineer
Chapter 15 interview drill
Interview questions: Temporal context and action chunking
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A policy predicts 50 actions at once but performs poorly when objects move. Distinguish prediction from execution horizon, explain padding and overlapping chunks, and design a test that finds the responsiveness boundary.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What is the difference between prediction horizon and execution horizon?
Prediction horizon is how many future actions the policy outputs; execution horizon is how many of those actions run before a new observation replaces or revises the plan.
Q2Why does a near-terminal action chunk need a mask?
The episode may end before the full horizon, so placeholders after the terminal point must not be trained as if the demonstrator issued them.
Q3What evidence should decide a chunk length?
Frozen rollouts measuring task success, response to changes, smoothness, inference and queue timing, stale-action exposure, and failures—not offline loss alone.