Chapter 15

Turn demonstrations into measured, bounded robot policies

Build imitation-learning policies without confusing supervised fit with autonomous skill. This chapter defines the observation-to-action contract for behavior cloning, traces why a learner creates unfamiliar states, applies DAgger-style expert relabelling in a safe simulator or paper world, compares temporal context with action chunks, opens ACT far enough to understand its sequence and latent-variable contracts, separates offline validation from closed-loop rollout evidence, constructs a deadline-aware inference loop, and finishes with a frozen comparison whose success, latency, interventions, and failures can be reproduced.

Before you start

Complete Week 13's tensor, Dataset, DataLoader, grouped-split, training-loop, checkpoint, reproducibility, and model-card work.
Complete Week 14's episode, trajectory, teleoperation, action-space, timestamp-alignment, schema, split, quality-review, and dataset-card work.
Bring a small non-sensitive demonstration table or a synthetic equivalent with episode IDs, timestamped observations, teacher actions, reset conditions, and terminal outcomes.
Keep every hands-on exercise in a paper world, a bounded simulator, or an already commissioned platform under qualified supervision; collecting expert labels does not authorize unsafe autonomous motion.

By the end

Specify behavior cloning as supervised learning over aligned observation-action examples, choose a loss that matches the action contract, and state exactly what low held-out loss does and does not prove.
Explain covariate shift and compounding error, then run a small DAgger-style loop that records learner-visited states, queries an expert safely, aggregates labelled data, retrains, and evaluates each resulting policy independently.
Distinguish observation history from an action chunk, calculate chunk tensor shapes and execution timing, and select a replanning or overlapping-chunk rule that preserves responsiveness.
Trace ACT training and inference through image and state observations, future-action targets, a conditional latent variable, transformer sequence prediction, reconstruction and KL terms, and bounded chunk execution.
Design offline and closed-loop evaluation as separate evidence layers with frozen episodes, initial conditions, success rules, intervention labels, latency measures, and failure stages.
Implement a policy loop that resets episode state, checks observation freshness, reproduces preprocessing, measures inference latency, validates and bounds decoded actions, rejects stale output, and enters a task-specific safe fallback.
Compare two policy checkpoints on the same predeclared suite and make a decision from paired trial evidence rather than a preferred demo, one loss value, or metrics chosen after seeing results.

The field story

Teach the handover policy to recover from the crooked approach

Week 14 froze handover-v1, including the clean places and the uncomfortable evidence: a gripper drifting left, a receiver-not-ready abort, and a late operator takeover. This week consumes that exact release rather than inventing a friendlier dataset. A behavior-cloning baseline may imitate familiar starts while creating new states after one small error. The mission follows that crooked approach through supervised loss, safe expert relabelling, temporal context, ACT-style chunks, closed-loop evaluation, and a deadline-aware runtime.

Two checkpoints will face the same frozen simulator cases. Neither earns trust from low validation loss, a smooth demo, or a preferred architecture. You will preserve teacher action, learner proposal, executed command, intervention, latency, guard result, and terminal outcome separately. When a stale prediction or oversized action appears, the runtime must reject it and enter the task-specific fallback. The spiral from Week 14 is deliberate: dataset contracts become policy contracts, and every rollout failure points back to evidence that can be repaired without contaminating final evaluation.

Why this chapter now: The course now has a traceable demonstration release and a correct training workflow, allowing imitation behavior and closed-loop distribution shift to be studied without guessing data semantics.
Ignore for now: Do not train a foundation model, run open-ended reinforcement learning, claim broad autonomy, or use powered hardware. Compare small imitation policies on one bounded simulated handover; keep the MDP/RL contrast to a paper model and deterministic simulator.
This unlocks: A measured policy runtime and evaluation discipline support later VLA adaptation, edge deployment, safety harnesses, and capstone training.
Proof you will leave with: Provide policy input/action contracts, loss and component errors, a bounded imitation-versus-RL decision map with state, action, transition, reward, and reward-hacking counterexample, DAgger lineage, chunk timing, ACT tensor trace, frozen offline and rollout manifests, runtime deadlines, fallback tests, paired trial ledger, interventions, and exact checkpoint identities.

Environment contractRepository-supported Node.js 22.13.0 or newer runs the fixed-suite comparator. Policy labs consume the learner-created handover-v1 release from Week 14 in a learner-pinned environment, optionally using ROS 2 Jazzy with Gazebo Harmonic for disconnected simulation; this repository does not bundle those fixtures, checkpoints, or simulator workspace.

Compatibility boundary: Policy processors, feature schemas, checkpoint formats, action statistics, and inference APIs can change across library revisions. Pin the actual implementation and verify load, reset, output shape, decoder, and guard behavior before rollout.
Smoke check: Run node week-15-compare-handover-policies.mjs; then load each chosen checkpoint in a fresh process, replay one golden observation, and verify reset state, decoded action, deadline decision, and fallback result.
Contract reviewed: 2026-07-25
Runtime evidence: The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
Drift risk: high

Today in the field story

One problem, then the next

Load handover-v1 without changing its contracts. Behavior cloning learns from teacher actions; an MDP instead names state, action, transition, and reward, while reinforcement learning improves from simulated return. Draw a decision map: choose imitation when demonstrations define the task, and consider bounded simulation-only RL when a valid reward and exploration envelope exist. Plant a reward for speed alone and show how collision can become a high-scoring hack.

Why now: Behavior cloning is the simplest measurable use of the frozen demonstration pairs.
Ignore today: Ignore autonomous claims and complex architectures; establish one transparent supervised baseline.
Unlocks next: A policy whose closed-loop distribution failures can be observed.

Understand

Build the physical picture first

Behavior cloning is a supervised translator: it receives a precisely defined robot observation and predicts the teacher action paired with that same physical moment.

Behavior cloning turns demonstrations into labelled examples. Each input is an observation available to the policy, such as a camera image, joint positions, gripper state, and task text. Each target is the teacher action that belongs to that observation under a declared timing rule. A model learns parameters that reduce a supervised loss between its prediction and the recorded target. It is not learning from reward, searching the environment for a plan, or discovering which result is safe. It is fitting the action choices represented in the demonstrations.

The observation-action pair must have one physical meaning before it becomes a tensor. If an image timestamp is 2.000 seconds and the action at 2.050 seconds reflects the operator's response to that image, the alignment rule must preserve that causal lag. Joint targets in radians cannot be silently mixed with measured joint state, degree values, velocity commands, or Cartesian positions. Normalization statistics come from the training split and must be saved with the checkpoint, because a correctly shaped but differently scaled value can produce a physically wrong command.

The loss expresses which prediction differences training tries to reduce. Mean squared error penalizes larger numeric errors strongly and can be suitable for one continuous action baseline; mean absolute error treats error size linearly; a categorical loss may suit a declared discrete action. No loss knows that two joints have different safe ranges or that averaging two valid ways around an obstacle may point into it. Report error per action component and in physical units after de-normalization, not only one convenient training scalar.

A held-out demonstration score tests examples drawn from the recorded distribution. During a rollout, the policy chooses actions that change the next observation. One small error can therefore create a state absent from the demonstrations, where the policy has no taught recovery. This is why low validation loss is useful evidence about imitation on recorded examples but is not evidence of task success, collision avoidance, timing fitness, or recovery. Those claims need bounded closed-loop trials with frozen resets and visible failures.

Words you need

Name each idea precisely

Behavior cloning: Supervised learning of a policy from observation-action pairs supplied by a demonstrator.
Physical example:
A model receives a picture and joint state from a block-sorting attempt and predicts the operator's next bounded joint target.
Policy: A function that maps the observations and optional context available now to an action proposal.
Physical example:
A policy maps front-camera pixels plus six joint positions to six target positions for the next control interval.
Supervised loss: A numerical rule comparing a policy prediction with the paired teacher target so training can adjust model parameters.
Physical example:
Action errors of 0.10 and -0.10 radians have a two-value mean squared error of 0.01 radians squared.
Action target: The teacher command selected as the correct output for one aligned observation under a declared representation and timing rule.
Physical example:
The target may be the leader arm's joint positions 50 milliseconds after the matching camera exposure, not the follower's later measured positions.
Closed-loop rollout: An episode in which the policy's action affects the world and therefore changes the observations the policy receives next.
Physical example:
A steering prediction moves a simulated rover off centre, so its next camera image is different from a recorded teacher-centred view.

Visual model

See the relationship

Swipe the technical canvas horizontally on a small screen.

Day 99 · Concept — Observation is current and decoded action stays inside the safe envelope. Measured anchor: 1 cm deviation.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalSupervised loss, rollout success, and the RL boundary

Low offline imitation error does not guarantee closed-loop recovery, while reinforcement learning adds a reward-driven decision process that can optimize unintended shortcuts.

MSE: average squared normalized action-component errorUnit: unitless
S/N: successful trials divided by all frozen trialsUnit: probability or percent
G = Σγᵗrₜ: discounted return used by an RL objectiveUnit: reward units

For one normalized, unitless action component, the errors are [1, −1, 2]. Square them: [1,1,4].
Average: MSE = (1+1+4)/3 = 2. Never average raw metres, radians, and discrete gripper values as though they share one unit.
Also run the policy: 16 successes from 20 frozen trials is 80%. If a later sim-only RL experiment is proposed, declare its state, actions, transitions, reward, discount, safety constraints, and reward-hacking tests before comparing the same suite.

Programmer analogy

A unit test can pass while the full user flow fails. RL additionally resembles optimizing a business KPI: a system may improve the number while violating the real intent.

What is success rate for 18 successes in 24 trials?

18/24 = 0.75 = 75%.

For two scalar action errors $0.1$ and $-0.1$ ,

\operatorname{MSE}=\frac{0.1^2+(-0.1)^2}{2}=0.01.

A rough independent-step model gives $p^N=0.98^{20}\approx0.668$ . Both are diagnostics; closed-loop rollout success remains the real test.

Calculate and interpret one tiny cloning loss

A two-joint policy is evaluated on three held-out observation-action pairs. Teacher actions are [0.20, 0.50], [0.00, 0.40], and [-0.20, 0.30] radians. Predictions are [0.30, 0.40], [0.10, 0.40], and [-0.10, 0.20] radians.

Verify the contract: each prediction and target has ordered axes [joint_1, joint_2], uses radians, refers to the same command type, and is aligned to the corresponding observation.
Subtract target from prediction to get six errors: [0.10, -0.10], [0.10, 0.00], and [0.10, -0.10] radians.
Square the six errors to get [0.01, 0.01, 0.01, 0.00, 0.01, 0.01] radians squared.
Add the squared errors to obtain 0.05, then divide by six action values: MSE = 0.05 / 6 ≈ 0.0083 radians squared.
Report component evidence too: joint 1 has three errors of 0.10 radians, while joint 2 has errors -0.10, 0.00, and -0.10; the average hides that pattern.
Keep the conclusion narrow: the checkpoint approximately matches these recorded targets; no rollout, recovery, latency, limit, or task-success claim has been tested.

Result

The toy held-out MSE is about 0.0083 radians squared, with a repeated +0.10-radian bias on joint 1 that deserves diagnosis.

What this proves

A loss becomes useful only after its axes, timing, units, denominators, and evidence boundary are readable.

Physical examples

Where this appears in real life

Teacher cards for a paper sorter

Draw eight block layouts on cards. On the back, write the teacher's next move as left, right, grasp, release, or stop. Let a learner rule view only the front and predict the matching move.

Look for:

The cards demonstrate supervised observation-to-action matching, while any layout not drawn on a card exposes the absence of taught behavior rather than a mysterious model failure.

Two paths around one cup

Lay arrow sequences that pass a cup on its left and right. At the same-looking start state, one demonstration begins with -1 and another with +1 on the sideways action axis.

Look for:

A squared-error predictor can average the targets to 0 and head at the cup; a small numeric loss does not guarantee that the mean action is physically valid.

Hands-on exercise

Make the idea observable

Use a spreadsheet or a short CPU-only program with twelve synthetic paper-sorter states and discrete or two-value continuous actions. Do not connect predictions to hardware.

Write the observation schema, action representation, units, component order, timestamp pairing rule, allowed range, and reset condition before creating a model input.
Split whole paper episodes into training and held-out groups; verify that no frame or near-duplicate layout from one episode crosses the boundary.
Create a nearest-card or tiny regression baseline, fit only the training pairs, and save every preprocessing value beside its rule or checkpoint.
Calculate held-out error overall and per action component, then inspect the largest three errors against their cards instead of reporting only an average.
Roll the policy through one familiar layout and one shifted layout on paper, letting each predicted move create the next state; record every state, action, and terminal result.
Write two separate conclusions: what the offline result supports and what the two closed-loop paper episodes reveal but cannot prove about a physical robot.

Observe

The model can reproduce individual teacher labels while its own sequence enters an unseen card or amplifies a small directional bias.

Done when

A reviewer can reconstruct every aligned pair, recalculate the loss and component errors, confirm the grouped split, and see that rollout evidence is reported separately.

Build today

Train behavior-cloning and ACT-style policies on the same task and compare rollout behavior.

Evidence to save

DONE when the learning log explains “Behavior cloning and supervised policy learning” in five precise points and a checked example produces the predicted output.

Common mistakes

Catch the wrong mental model

Wrong

Calling behavior cloning reinforcement learning because the robot eventually attempts a task.

Better

Behavior cloning fits demonstrator-labelled actions with supervised learning; task interaction during later evaluation does not add a reward-learning objective.

Wrong

Pairing every observation row with the action at the same row number without checking clocks or operator delay.

Better

Define and test the causal timestamp-alignment rule so each target is the teacher response intended for that observation rather than merely a nearby record.

Wrong

Reporting one normalized MSE as proof that actions are accurate and safe.

Better

De-normalize predictions, report component errors with units and limits, inspect worst cases, then use independent guards and closed-loop trials for behavior claims.

Job connection

How this becomes employable evidence

Turn a demonstration schema into an explicit policy input and action target, build preprocessing and output assertions around the training code, and review per-joint physical errors before a checkpoint is allowed into a bounded rollout harness.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Deployment, Integration & Validation Engineer

Chapter 15 interview drill

Interview questions: Behavior cloning and supervised policy learning

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Your behavior-cloning checkpoint has very low validation MSE but repeatedly misses the task. Trace observation-action alignment, units, normalization, split leakage, multi-modal targets, and closed-loop distribution shift in the order you would test them.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What exactly is the labelled example in behavior cloning?

Model interview answer

It is an observation and the demonstrator action paired to it under a declared schema, representation, unit, and timing rule.

Q2Why can averaging two teacher actions be physically wrong?

Model interview answer

When the actions are two distinct valid modes, such as passing an obstacle on opposite sides, their numeric mean can lie between the modes and point into the obstacle.

Q3What does low held-out action loss fail to establish?

Model interview answer

It does not establish closed-loop task success, recovery from learner-created states, deadline fitness, safe action execution, or performance outside recorded conditions.

Chapter references

LeRobot — Imitation Learning on Real-World RobotsMaintainer workflow connecting a recorded robot dataset to ACT training, checkpoint resume behavior, policy selection, and controlled policy use on the matching robot configuration.
LeRobot — Adding a PolicyOfficial policy interface for episode reset, per-step action selection, action-chunk prediction, training loss, padded-action masks, feature validation, preprocessing, postprocessing, and evaluation integration.
Ross, Gordon, and Bagnell — DAggerPrimary paper for the sequential-distribution problem and the DAgger loop: roll out the current or mixed policy, ask the expert to label visited states, aggregate those examples, retrain, and select using validation.
Zhao et al. — Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwarePrimary ACT paper for future-action sequence prediction, effective-horizon reduction, overlapping chunks and temporal ensembling, the conditional-VAE training path, compounding-error motivation, ablations, and rollout evidence.
LeRobot — Policy DeploymentMaintainer deployment contract for a real policy loop, control frequency, synchronous versus queued chunk inference, observation-to-action execution, recording, reset behavior, and latency-sensitive runtime choices.