Chapter 23

Collect and freeze a task-shaped capstone dataset

Turn one bounded tabletop or mobile-robot capstone into a versioned first dataset whose meaning, operating envelope, operator authority, timing, coverage, splits, and limitations can be reconstructed. The chapter begins with observable physical success and controlled variation, then integrates a simulation-first Flutter or Web console with leased dead-man control, independently enforced command limits, visible latency, cancellation, and synchronized recording. It collects a deliberately balanced first batch, preserves and classifies failures, repairs measured coverage holes, isolates whole scene groups across train, validation, and held-out partitions, and freezes dataset v1 with an immutable manifest, card, reviewed examples, checksums, and denominator-backed quality metrics. More episodes are not accepted as a substitute for safer control or clearer evidence.

Before you start

Complete the earlier robot-data schema, episode timing, dataset-quality, safe HMI, action cancellation, hardware bring-up, calibration, limits, watchdog, and acceptance-gate chapters. This week integrates those contracts into one collection system rather than weakening them for faster recording.
Bring one harmless simulated task or an already commissioned low-risk platform, a repeatable reset, a versioned robot and camera configuration, a no-motion startup state, and an approved safe-stop procedure. Learner exercises remain in simulation unless the responsible qualified people have approved the real workspace and supervision.
Be able to distinguish operator intent, accepted bounded command, measured robot state, camera observation, task outcome, review disposition, and dataset inclusion. These are separate records and none may be inferred merely from a responsive interface.
Bring a local storage location with enough capacity for raw episodes, a privacy and rights boundary for every camera view, and a way to preserve raw attempts immutably while release manifests select reviewed episodes without deleting inconvenient evidence.

By the end

Define a capstone episode through allowed objects, initial-state predicates, terminal success predicates, timeout, intervention policy, abort reasons, reset proof, units, tolerances, and a denominator-backed evaluation claim.
Design bounded variation across scene, object instance, lighting, camera, start state, and operator without creating accidental correlations, unsafe combinations, or an unrepeatable collection schedule.
Integrate a Flutter or Web teleoperation path whose UI dead-man events stop renewal, while authentication, exclusive authority, freshness, command limits, cancellation, and timeout are enforced outside the browser and verified against measured simulated state.
Record camera observations, robot state, raw operator intent, accepted command, timing, control events, task identity, and terminal labels under one episode identity and a declared clock-alignment contract.
Collect and review a first batch by planned condition cells, retain raw failures, classify defects and outcomes, and target additional collection at measured deployment-relevant coverage gaps.
Assign whole related scene groups to train, validation, and held-out scene partitions, keep derived data and fitted statistics on the correct side of each boundary, and prove split disjointness.
Freeze an immutable dataset v1 whose manifest, schema, splits, card, examples, metrics, code and calibration identities, checksums, rights decision, known limitations, and reproduction checks agree.

The field story

Episode Foundry Dataset v1

Bench Zero hands the Episode Foundry a mock robot, calibration identity, command limits, and a strict no-motion startup contract. The foundry's mission is to create Dataset v1 for one bounded capstone task without confusing a responsive teleoperation screen with safe collection or a large episode count with useful coverage. Every episode begins from declared predicates, ends in an observable label, and joins camera observations, robot state, operator intent, accepted commands, timing, interventions, cancellation, and reset evidence under one identity. Raw failures remain immutable even when the release manifest excludes them.

The dataset grows through controlled variation rather than random accumulation. You define the task and condition cells, then connect a simulation-first Web or Flutter console whose dead-man renewal can expire outside the browser. The first batch deliberately covers planned cells, and review converts failures into a taxonomy instead of deletion. Additional collection targets measured holes. Whole related scene groups cross only one of train, validation, or held-out boundaries, and fitted statistics stay on the correct side. Dataset v1 freezes with checksums, card, examples, rights decisions, quality denominators, and reproduction proof, becoming the sealed input to Week 24.

Why this chapter now: Week 22 established mock hardware, calibration, limits, and collection safety prerequisites. The capstone now needs task-shaped data whose meaning and boundaries survive training.
Ignore for now: Do not train a model, tune on held-out scenes, delete failed raw attempts, infer safety from a dead-man button, or collect powered data without approved platform controls.
This unlocks: A sealed Dataset v1 that Week 24 can use for a falsifiable baseline, training, export, edge profiling, and one untouched final evaluation.
Proof you will leave with: Task and reset contracts, condition-cell plan, teleoperation lease traces, synchronized episode manifests, raw/reviewed dispositions, failure taxonomy, coverage table, group-disjoint split proof, checksums, examples, card, and reproduction output.

Environment contractrepository-supported Node.js 22.13.0 or newer, synthetic episodes, local files, and a simulated or mock robot contract; no camera upload, remote service, package install, or powered collection is required.

Compatibility boundary: The starter validates split identities only. A real Flutter/Web, ROS 2, LeRobot, camera, or robot collection stack must separately verify clocks, calibration, authentication, exclusive authority, limits, storage, privacy, rights, and finalization behavior.
Smoke check: Run node week-23-dataset-v1-manifest.mjs; confirm the clean scene groups pass, the planted train-to-held-out leak is rejected, and Dataset v1 remains held until the full manifest is frozen.
Contract reviewed: 2026-07-25
Runtime evidence: The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
Drift risk: medium

Today in the field story

One problem, then the next

The Episode Foundry writes its job ticket before recording anything. The capstone task names allowed objects, initial predicates, success and partial-success predicates, timeout, intervention policy, abort reasons, reset evidence, units, and tolerances. One video-worthy movement is not a dataset objective. Every future episode must answer whether the declared physical or simulated outcome occurred and why the attempt belongs in the denominator, giving Week 24 a target whose performance can be measured rather than narrated.

Why now: Data quality begins with an observable task and terminal labels.
Ignore today: Do not select a policy or optimize episode count.
Unlocks next: A stable episode contract and denominator for collection and evaluation.

Understand

Build the physical picture first

A capstone task is a marked physical game board with a start photograph, legal moves, a finish photograph, a timer, and explicit reasons to stop.

Start with the scene a camera and reviewer could inspect. Name the robot, tool, object population, workspace, permitted contact surfaces, initial robot mode, object start region, target region, and reset condition. Then state one small useful transformation, such as moving a named foam block from a pickup rectangle into a bin. “Sort objects” is a project theme, not an episode contract, because it leaves object identity, quantity, placement tolerance, completion, timeout, and forbidden behavior undecidable.

Write success as simultaneous observable predicates rather than a human impression. A tabletop episode might require object_in(target_zone, tolerance_mm=15), gripper_empty=true, robot_at(park, tolerance_deg=3), collision_count=0, and elapsed_s <= 25. Name the sensor or review evidence for each predicate and what happens when a measurement is missing. A recorder’s success button can propose a label, but synchronized terminal evidence and a review rule decide whether that label is defensible.

Bound the episode on both sides. Entry predicates prevent collection from beginning with an already moved object, stale calibration, occupied workspace, missing camera, or active fault. Exit states include success, task failure, operator abort, safety stop, infrastructure invalidation, and timeout; these outcomes must not collapse into one Boolean. Define reset proof separately, because placing the object back approximately can silently change difficulty from one attempt to the next.

Separate the dataset objective from the later policy claim. The collection objective can be “capture reviewed demonstrations across the declared start-state grid,” while the policy acceptance claim may be “at least 16 of 20 held-out trials satisfy every terminal predicate without intervention.” Demonstration quality, dataset coverage, and autonomous success are related but different denominators. Freezing those definitions now prevents a disappointing model from causing the task rules to be softened afterward.

Words you need

Name each idea precisely

Episode contract: The versioned definition of valid entry state, allowed task, terminal outcomes, timeout, evidence, intervention rule, and reset for one recorded attempt.
Physical example:
A block-transfer episode begins only with the arm parked and block fully inside the start rectangle, and ends when the block is verified in the bin, the timer expires, or an abort occurs.
Success predicate: A measurable statement that must be true, with named tolerance and evidence source, for an episode to count as task success.
Physical example:
The whole foam cube lies inside the taped target boundary with no edge closer than 15 mm to the line.
Entry predicate: A condition that must be freshly verified before recording and command authority can begin.
Physical example:
The gripper is open, the joint state is fresh, the workspace is clear, and the cube is inside its assigned start cell.
Terminal outcome: One explicit reason an episode ended, kept richer than a success-versus-failure flag.
Physical example:
An attempt ends as operator_abort after a loose camera mount is noticed, rather than being mislabeled as a failed grasp.
Reset proof: Recorded evidence that the next episode starts from the declared robot, object, scene, and authority state.
Physical example:
A reset image and joint-state snapshot show the cube in start cell C2 and the arm within three degrees of park before episode 18 opens.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalDataset coverage and independent scenes

More frames from the same easy scene do not equal broader robot experience.

N_scene: independent physical setupsUnit: scenes
coverage: tested condition cells representedUnit: percent
failure rate: failed frozen trials divided by all trialsUnit: percent

Your matrix has 3 object positions × 2 lighting levels = 6 condition cells.
Data covers 4 cells, so coverage is 4/6 = 66.7%.
Collect the two missing cells before duplicating the easiest one; split by whole scene, not adjacent frames.

Programmer analogy

It is test-case coverage, not line coverage: the unit is a distinct physical condition.

Seven of ten condition cells are covered. What is coverage?

7/10 = 70%.

Under one frozen condition, task success is

r_{\mathrm{success}}=\frac{S}{N}=\frac{8}{10}=80\%.

Condition coverage is $c_i=n_i/N$ . Report both, because a high success rate on one easy condition does not prove broad task readiness.

Turn a vague pick-and-place idea into one frozen episode contract

The idea is “teach the arm to put a sponge in a bowl.” The simulated arm has one gripper, two sponge instances, an overhead camera, joint state, a 30 cm square workspace, and a named park pose.

Choose one task identity: task_id=place_sponge_v1; bind the episode to sponge instance S1, pickup region P, bowl region B, gripper G1, camera calibration C7, and robot configuration R12 instead of using the words “the sponge” alone.
Define entry predicates: arm within three degrees of park, gripper open, S1 completely inside P, S2 outside the active workspace, camera and joint-state ages below 100 ms, no active fault, and one exclusive operator lease available.
Define success predicates: S1 remains inside bowl B for two seconds, gripper is empty, arm returns to park, no forbidden-zone or collision event occurred, and elapsed time is at most 25 seconds.
Enumerate mutually understandable terminal outcomes: success, task_failure, operator_abort, safety_stop, invalid_recording, and timeout; attach the decisive event and final synchronized observation to each episode.
Write reset proof as one image, object-region result, fresh joint snapshot, empty-gripper state, cleared stop state, and new episode identity; reject collection if any reset check is unknown.
Freeze an evaluation statement before collection: a later policy passes only if at least 16 of 20 held-out episodes satisfy every success predicate with zero human intervention, while all 20 remain in the denominator.

Result

The broad idea becomes a reviewable contract with named entities, six entry checks, five simultaneous success checks, six terminal outcomes, repeatable reset evidence, and an evaluation denominator that cannot be edited after seeing results.

What this proves

A useful capstone begins when two reviewers can label the same attempt from recorded evidence without asking what the learner intended.

Physical examples

Where this appears in real life

Foam block into a divided tray

A small simulated arm begins parked beside two visually different foam blocks and must place only block B-04 into tray compartment T2 without touching the other block.

Look for:

Identity, start region, allowed contact, target tolerance, untouched-object rule, timeout, park state, and terminal evidence are all decidable before the first demonstration.

Rover delivers one empty carton

A simulated differential-drive rover begins inside a taped start box, carries an empty carton, and must stop completely inside a delivery bay while a narrow exclusion strip remains untouched.

Look for:

The contract defines footprint containment, measured zero velocity, carton retention, zone contact, time limit, abort states, and the exact reset pose rather than accepting “looks delivered.”

Hands-on exercise

Make the idea observable

Use a simulator or paper scene with one robot outline, two harmless objects, start and target regions, a clock, and a visible park state. Do not connect the exercise to powered hardware.

Use the chapter’s official LeRobot references to isolate this collection task, then write five operating rules and the one-sentence physical transformation with every robot, object, tool, region, and sensor identity required to judge it.
Draw the valid start scene, then write five to eight entry predicates with units, freshness limits, and a named evidence source for each.
Draw the successful finish scene and convert it into simultaneous terminal predicates, including robot safe state, forbidden events, tolerance, and timeout.
Create a terminal-outcome table that distinguishes task failure, operator abort, safety stop, invalid recording, and timeout from success.
Implement one minimal simulated or paper executable, then perform three walkthroughs: nominal success, object starting outside its region, and camera evidence missing at the apparent finish.
Have another person label the three attempts using only the contract; revise every sentence that still requires guessing, then assign the contract a version and checksum.

Observe

Most ambiguity appears at entry, terminal evidence, or reset rather than in the central motion, and a single success flag hides several operationally different endings.

Done when

A second reviewer reaches the same outcome for all three walkthroughs, every predicate names evidence and tolerance, and no attempt begins from an unknown reset state.

Build today

Collect, review, and version real demonstrations for one useful tabletop or mobile task.

Evidence to save

DONE when the learning log explains “Define the capstone task and measurable success” in five precise points and a checked example produces the predicted output.

Common mistakes

Catch the wrong mental model

Wrong

Calling “pick and place reliably” a measurable capstone task.

Better

Name the entity, start and target regions, tolerances, timeout, allowed contacts, forbidden events, terminal robot state, intervention policy, and evidence source for every required predicate.

Wrong

Letting the operator’s success button become ground truth.

Better

Store the button as an annotation, then derive the reviewed outcome from synchronized terminal observations, robot state, event history, and the frozen rubric.

Wrong

Removing invalid or aborted attempts from the denominator without recording them.

Better

Preserve every opened episode in the raw ledger, give it one explicit outcome and reason, and publish both collection and curated-release denominators.

Job connection

How this becomes employable evidence

Translate a customer manipulation or AMR objective into versioned data-collection and policy-acceptance contracts that join ROS 2 state, camera evidence, intervention rules, safety events, and denominator-backed trial outcomes.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer

Chapter 23 interview drill

Interview questions: Define the capstone task and measurable success

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Define a capstone episode for placing one object into a bin. State entry and success predicates, reset proof, timeout, invalid versus failed outcomes, and the evidence required before you would accept a success label.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why is “the object reached the bowl” not yet a complete success definition?

Model interview answer

It omits object identity, containment tolerance, persistence, allowed contact, timeout, intervention, terminal robot state, and the evidence used to judge the claim.

Q2What is the difference between `task_failure` and `invalid_recording`?

Model interview answer

task_failure is a validly observed attempt that missed the task criteria; invalid_recording means evidence or protocol was defective enough that the task outcome cannot be trusted.

Q3Why freeze the autonomous evaluation denominator before training?

Model interview answer

It prevents the task, threshold, or counted trials from being changed after results are known, which would turn evaluation into retrospective storytelling.

Chapter references

Hugging Face LeRobot — Processors for Robots and TeleoperatorsOfficial LeRobot guidance for separating teleoperator, dataset, and robot action spaces; declaring feature transformations; recording frames against the dataset feature contract; and beginning custom robot pipelines with tight bounds and simulation.
Hugging Face LeRobot — Getting Started with Real-World RobotsOfficial end-to-end reference for keeping robot and teleoperator calibration identities consistent across teleoperation, recording, visualization, training, and evaluation, and for structuring demonstrations as repeatable episodes.
Hugging Face LeRobot — LeRobotDataset v3.0Official dataset-format reference for incremental frame and episode recording, metadata, Parquet and video finalization, and the requirement to finalize writers before treating a v3 dataset as complete and loadable.
ROS 2 Documentation — ActionsOfficial ROS 2 explanation of long-running goals with feedback and cancel or preempt behavior, used to keep a cancel request, cancel processing, terminal result, and observed physical stop as different states.
Hugging Face LeRobot — Human-In-the-Loop Data CollectionOfficial LeRobot description of recording interventions, recovery motions, corrections, and autonomous segments so later coverage work can preserve failure context rather than training only on ideal states.