Chapter 15 · Turn demonstrations into measured, bounded robot policies
Today in the field story
One problem, then the next
Roll the baseline in the simulator until it reaches the crooked state absent from clean demonstrations. Keep learner proposal, executed command, supervisor action, intervention, and abstention distinct. Ask the expert for a label only when the saved observation is attributable and safe, aggregate that state under a new lineage, retrain, and evaluate separately. The correction loop targets learner-visited states without rewriting handover-v1 or borrowing final cases.
- Why now
The baseline reveals covariate shift that more original-route demonstrations do not directly repair.
- Ignore today
Ignore online learning on hardware; run one bounded DAgger-style correction cycle.
- Unlocks next
A corrected candidate and evidence about recovery-state coverage.
Understand
Build the physical picture first
DAgger is a supervised correction cycle: let the learner reveal states it actually visits, ask an expert what should happen there, add those labels, and retrain.
Ordinary behavior cloning trains on observations visited by the demonstrator. During autonomous use, the learner's small action errors alter the scene, so later observations come from the learner's own state distribution. This mismatch is covariate shift: the input distribution changes even if the action definition stays the same. Errors can compound because an unfamiliar state produces another poor action, which creates a still less familiar state. The rough expression p^N illustrates fragility only under independent equal-probability decisions; robot decisions are linked, so actual rollouts remain necessary.
DAgger, short for Dataset Aggregation, directly collects labels on states induced by successive learner policies. In the original loop, an iteration rolls out a mixture of the expert and current learned policy, queries the expert for the action they would choose at each visited state, adds those labelled states to the accumulated dataset, and trains the next policy on the aggregate. The expert action used as a label need not be the action that was executed at that moment. Keep executed controller, expert label, learner proposal, and any intervention as separate fields.
An expert or oracle must be able to label recovery states correctly. A hurried operator takeover is not automatically a clean DAgger label: the takeover may begin late, contain communication delay, or reflect emergency motion rather than the preferred action from the recorded state. Define when labels are requested, how their timestamps align, whether the expert or learner controls motion, and how disagreement and abstention are stored. Querying an unsafe or ambiguous state should produce an abstain, abort, or quarantine result rather than a fabricated target.
DAgger changes the data and the policy every round, so every round needs lineage and independent evaluation. Record the starting checkpoint, mixture rule or intervention policy, rollouts, learner-visited states, expert identity, labels, aggregation recipe, training configuration, and resulting checkpoint. Do not declare improvement from a larger dataset or a falling aggregate loss. Freeze an evaluation suite that is not added to training, run each candidate on it, and report task success, interventions, new failure states, and safety stops.
Words you need
Name each idea precisely
- Covariate shift
A change in the distribution of observations presented to the policy between training examples and later use.
Physical example:A line follower trained on centred camera views begins seeing the line at the image edge after its own steering error.
- Compounding error
A sequence effect in which one prediction error changes a later input and makes additional errors more likely or more severe.
Physical example:A gripper approaches five millimetres too far left, misses the handle, then receives views of an empty grasp never present in successful demonstrations.
- DAgger
An iterative imitation-learning method that labels states visited by the learner or an expert-learner mixture, aggregates them with earlier data, and retrains.
Physical example:After a paper rover drifts into an edge cell, a teacher labels the turn that would recover from that exact cell and the example joins the next training set.
- Expert query
A request for the action an authorized, qualified demonstrator would choose from a recorded learner-visited state.
Physical example:A simulator pauses on an off-centre view and asks the supervisor to mark turn-right, turn-left, continue, stop, or abstain.
- Dataset aggregation
Appending newly labelled learner-visited examples to a traceable accumulated dataset rather than replacing the original demonstrations.
Physical example:Round 0's 40 teacher states and Round 1's 12 recovery labels form a versioned 52-example training view with original round IDs preserved.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalSupervised loss, rollout success, and the RL boundary
Low offline imitation error does not guarantee closed-loop recovery, while reinforcement learning adds a reward-driven decision process that can optimize unintended shortcuts.
- MSE
- average squared normalized action-component errorUnit: unitless
- S/N
- successful trials divided by all frozen trialsUnit: probability or percent
- G = Σγᵗrₜ
- discounted return used by an RL objectiveUnit: reward units
For one normalized, unitless action component, the errors are [1, −1, 2]. Square them: [1,1,4].
Average: MSE = (1+1+4)/3 = 2. Never average raw metres, radians, and discrete gripper values as though they share one unit.
Also run the policy: 16 successes from 20 frozen trials is 80%. If a later sim-only RL experiment is proposed, declare its state, actions, transitions, reward, discount, safety constraints, and reward-hacking tests before comparing the same suite.
A unit test can pass while the full user flow fails. RL additionally resembles optimizing a business KPI: a system may improve the number while violating the real intent.
What is success rate for 18 successes in 24 trials?
18/24 = 0.75 = 75%.
Under the stated independence approximation, the probability of remaining on-distribution for all steps is
Dependence between real policy errors means this is only an illustration.
Run two dataset-aggregation rounds on a paper rover
Round 0 contains 20 teacher-labelled centre states. A cloned policy succeeds on 6 of 10 frozen paper routes and enters four edge states. The expert can label all four edge states. After aggregation and retraining, the next policy succeeds on 8 of the same 10 route definitions in a fresh run.
Freeze the ten evaluation route definitions and keep their trial outcomes outside training; the fact that route definitions repeat does not permit their observed evaluation labels to enter the dataset.
Record Round 0 as 20 examples, its data version, policy checkpoint, and
6/10 = 60%observed evaluation success.Roll out the Round 0 policy on separate collection routes; for each edge state store the learner proposal, action actually executed, expert label, query time, and whether the expert abstained.
Validate the four expert labels, then aggregate rather than overwrite:
20 + 4 = 24labelled examples with source round and original identity.Train a new checkpoint from the declared 24-example view and do not select it from the frozen suite one trial at a time.
Run the complete frozen suite and report
8/10 = 80%, two remaining failure routes, interventions and stops; the observed increase is 20 percentage points, not proof of general superiority.
The aggregate grows from 20 to 24 traceable examples, while frozen-suite observed success changes from 60% to 80%; two failures and the small sample remain visible.
DAgger earns evidence by targeting learner-visited states and re-evaluating, not by calling every correction an improvement.
Physical examples
Where this appears in real life
Tape-road recovery labels
Draw a road with centre, left-edge, and right-edge cells. A teacher route contains only centre cells. Let a simple learner take one wrong move, then pause and ask a teacher to label the recovery move from the reached edge cell.
More repeats of centre driving do not teach the missing edge recovery; the useful new example comes from the learner-induced state and retains its round identity.
Spoon carrying with a supervisor
A child carries a dry spoon along floor marks while a supervisor names the move they should make whenever they step off the marked route; no liquid, robot, or risky takeover is involved.
The spoken correct move is a label for the off-route state. It must not be confused with the child's executed step or a late physical intervention.
Hands-on exercise
Make the idea observable
Use a paper grid, a simple browser simulation, or a local deterministic script. One person may act as the expert; do not use a powered robot or deliberately drive hardware toward a hazard.
Create teacher data from safe centre-path states, define left, right, forward, stop, and abstain labels, then train or write the simplest policy that imitates those labels.
Freeze five evaluation routes and five separate collection routes with start cell, obstacle cells, terminal rules, and seed or card identity.
Run the learner on collection routes, capture every visited state, and ask the expert for a label without erasing learner proposals, executed actions, or aborts.
Inspect and aggregate only valid aligned labels, preserving round, route, expert, query, abstention, and original-example identifiers.
Retrain one next-round policy, then run both checkpoints across the entire frozen evaluation suite with identical reset and scoring rules.
Publish a round table containing dataset size, newly covered states, success count, interventions, abstentions, failure cells, and the next collection decision.
The learner reveals recovery states absent from teacher-only routes, while some states remain too ambiguous for a forced label and should be stopped or quarantined.
Every new label traces to a learner-visited state, no evaluation outcome leaked into training, both policies completed the same suite, and remaining failures are named.
Build today
Train behavior-cloning and ACT-style policies on the same task and compare rollout behavior.
Evidence to save
DONE when “Covariate shift, DAgger, and human-in-the-loop corrections” runs from one documented command and the nominal plus boundary outputs are attached.
Common mistakes
Catch the wrong mental model
Calling extra demonstrations from the original perfect route DAgger.
DAgger's defining value is expert labelling of states induced by the learner or its expert-learner mixture; repeated original-route data does not address that distribution.
Saving only the expert correction and losing what the learner proposed or what the controller executed.
Store learner proposal, expert label, executed action, intervention state, timing, and round separately so training and safety behavior remain reconstructable.
Forcing the expert to label every unsafe or ambiguous state.
Provide abstain, abort, and quarantine outcomes; an invented target can teach dangerous behavior and hides a real coverage boundary.
Using the same corrected rollouts to train and to claim improvement.
Aggregate collection labels into training, but compare policy checkpoints on a separate frozen suite with unchanged resets and scoring.
Job connection
How this becomes employable evidence
Design a human-in-the-loop correction workflow that distinguishes policy proposals, executed commands, supervisor labels, takeovers, abstentions, and stops, then prove each aggregated checkpoint on an untouched rollout suite.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robot HMI / Control & Monitoring Engineer
Chapter 15 interview drill
Interview questions: Covariate shift, DAgger, and human-in-the-loop corrections
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Describe one safe DAgger iteration for a robot that drifts outside its demonstration data. Explain the control mixture, expert-query timing, dataset lineage, abstention path, retraining boundary, and independent evaluation.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What distribution problem is DAgger trying to address?
The learner visits observations created by its own actions, while ordinary cloning was trained mainly on observations visited by the expert.
Q2During a DAgger query, are expert label and executed action always identical?
No. The expert can label what should be done while a mixed controller, learner, stop, or intervention determines what was actually executed; record them separately.
Q3Why must each aggregated checkpoint face an untouched suite?
Training on newly corrected states can lower loss without improving closed-loop behavior, so only separate fixed trials reveal whether task performance and failures changed.