Chapter 23 · Collect and freeze a task-shaped capstone dataset

Today in the field story

One problem, then the next

Episode Foundry Dataset v1 closes like a manufactured lot. Manifest, schema, episode index, split groups, calibration and software identities, card, representative examples, condition metrics, exclusions, rights decisions, and checksums must agree. A clean process reloads the release and reproduces counts without consulting mutable working folders. Known timing, coverage, sensor, operator, and simulation limitations remain prominent. The sealed train and validation portions travel to Week 24; the held-out envelope stays closed until the experiment charter permits one final evaluation.

Why now: Training requires a stable data identity, not a folder that changes under the run.
Ignore today: Do not patch Dataset v1 in place or open the held-out envelope.
Unlocks next: A reproducible immutable input for the Policy Forge.

Understand

Build the physical picture first

Dataset v1 is a sealed evidence crate: the inventory, measurements, inspection marks, examples, ownership, limitations, and fingerprints must all describe the same contents.

Freezing means immutability plus reproducibility, not perfection. Separate raw capture, review annotations, release recipe, and frozen release. Dataset v1 receives a stable identity, schema version, exact episode and split manifests, recorder and protocol revisions, robot and calibration identities, and content checksums. Once published or used for a reported training run, those bytes and assignments do not change in place; a correction creates a traceable successor.

Write the dataset card from calculated artifacts. State task and episode contracts, hardware or simulation environment, teleoperation path, feature names and ordered semantics, units, frames, sampling and synchronization method, variation envelope, operators and sessions at an appropriate privacy level, counts and durations, split logic, quality rubric, exclusions, rights decision, intended use, unsupported use, limitations, known sparse cells, and contact or ownership channel. Every headline metric names its denominator and manifest.

Choose examples that expose the release rather than advertise it. Include at least one ordinary success, one hard but valid success, one excluded or quarantined case with reason, and one terminal or synchronization trace. Each example links camera, measured state, applied action, timing residual, task fields, outcome, disposition, and source episode without leaking a sealed held-out evaluation scene. A visualizer opening an example proves inspectability, not correctness, so automated and human checks remain separate.

Run a freeze gate from empty cache to loaded episode. Finalize writers, verify every referenced media and metadata range, validate schema and timestamps, reconcile counts and duration, prove split and lineage disjointness, recompute checksums, recalculate metrics, scan privacy and secrets, and reproduce the card tables. Store the exact verification command and result. If any gate fails, v1 remains a candidate; do not edit the report to match broken bytes.

Document what v1 cannot support. A small dataset may omit reflective objects, extreme yaw, alternate cameras, recovery behavior, or multiple operators. Those limitations define the next collection backlog and cap policy claims. Version v1.0.1 can repair documentation or labels only under a stated compatibility rule; changed episode membership, split groups, task semantics, or feature meaning normally requires a new release lineage. A frozen weak dataset is honest; a mutable “final” dataset is not evidence.

Words you need

Name each idea precisely

Frozen manifest: The immutable inventory of release episode, group, partition, disposition, source, and content identities used to reconstruct exactly what v1 contains.
Physical example:
Manifest M-v1 lists 24 training, 6 validation, and 6 held-out episode IDs with scene groups and SHA-256 content fingerprints.
Dataset card: Versioned documentation of task, collection, schema, provenance, quality, splits, rights, intended use, limitations, and measured release statistics.
Physical example:
The card states that dim-right scenes have only three release-ready examples and forbids a claim about reflective objects that were never collected.
Content checksum: A deterministic fingerprint used to detect whether a referenced file or manifest changed after the release was frozen.
Physical example:
Changing one Parquet byte causes its recorded SHA-256 value to disagree during verification.
Release candidate: A complete proposed dataset version that must pass all freeze gates before it earns the final immutable identity.
Physical example:
Candidate v1-rc3 loads locally but stays a candidate because one video range referenced by episode E28 is missing.
Known limitation: A measured or observed boundary that constrains appropriate use and future performance claims.
Physical example:
Only matte sponge objects under 180–420 lux were collected, so glossy objects and sunlight remain unsupported.
Release lineage: The traceable relationship among a frozen version and later corrected or expanded versions, including exactly what changed and why.
Physical example:
Dataset v2 descends from v1 but adds new object instances and scene groups while retaining v1 checksums and a machine-readable change log.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalDataset coverage and independent scenes

More frames from the same easy scene do not equal broader robot experience.

N_scene: independent physical setupsUnit: scenes
coverage: tested condition cells representedUnit: percent
failure rate: failed frozen trials divided by all trialsUnit: percent

Your matrix has 3 object positions × 2 lighting levels = 6 condition cells.
Data covers 4 cells, so coverage is 4/6 = 66.7%.
Collect the two missing cells before duplicating the easiest one; split by whole scene, not adjacent frames.

Programmer analogy

It is test-case coverage, not line coverage: the unit is a distinct physical condition.

Seven of ten condition cells are covered. What is coverage?

7/10 = 70%.

Required-field completeness is

C=\frac{N_{\mathrm{valid\ required}}}{N_{\mathrm{required}}}=\frac{9800}{10000}=98\%.

This metric cannot detect semantically wrong values or changed files, so retain checksums, schema validation, and visual review.

Reconcile and freeze one 36-episode v1 release

Release candidate RC3 selects 36 reviewed episodes from 42 raw attempts. Its split manifest assigns 24 train, 6 validation, and 6 held-out episodes across disjoint scene groups. Total selected duration is 540 seconds.

Reconcile membership: 24 + 6 + 6 = 36 selected episodes; separately report 36 / 42 = 85.7% release selection with the six excluded or quarantined raw identities and reason codes retained.
Recalculate duration as 540 / 36 = 15.0 s mean per selected episode while also storing distribution summaries and refusing to infer consistency from the mean alone.
Finalize Parquet and video writers, load RC3 from an empty cache, resolve every episode range and required feature, scrub selected examples, and fail if any terminal record or media reference is missing.
Verify schema order, units, frames, clocks, residual threshold, task and protocol versions, robot and calibration identities, and split inheritance for every derivative; compute checksums only after these bytes are final.
Run split tests showing no shared episode, scene group, source ancestry, or raw content hash across train, validation, and held-out, and confirm normalization provenance points only to the train manifest.
Generate card counts, coverage tables, quality rates, limitations, and example links directly from the frozen manifests; perform privacy, secret, license, and intended-use review before publication scope is decided.
Promote RC3 to dataset v1 only when a clean verification run reproduces all metrics and hashes; archive the command, result, manifests, rubric, code revisions, and change policy beside the immutable release.

Result

Dataset v1 contains 36 loadable reviewed episodes with reconciled 24/6/6 grouped splits, calculated metrics, immutable fingerprints, review lineage, inspectable examples, and limitations tied to actual coverage.

What this proves

The v1 label is earned when the bytes, manifests, examples, metrics, and written claims agree under a clean independent verification.

Physical examples

Where this appears in real life

A polished card points to a missing video

The release README reports 36 reviewed episodes, but an empty-cache verifier cannot open episode E28 because its camera range was never finalized into the referenced video shard.

Look for:

The release remains a candidate despite attractive documentation; finalization, referential integrity, load, playback, and count reconciliation must pass against the actual frozen files.

Quiet label edit changes a reported result

After training, someone changes one failed held-out episode to success inside the existing v1 folder and updates the summary percentage without publishing a successor.

Look for:

Checksums and immutable manifests detect the mutation; the correction needs review, change log, new identity, recalculated metrics, and an explicit effect on prior claims.

Hands-on exercise

Make the idea observable

Use the reviewed local simulation dataset and split manifest. Keep the release private if rights, privacy, credentials, or publication authority are unresolved.

Without looking at yesterday’s notes, recreate the freeze checklist, compare it with the actual contract, then create separate raw, review, and release manifests with candidate, schema, task, protocol, robot, calibration, recorder, and code revisions.
Finalize storage writers, then validate load, media decode, episode ranges, required fields, timestamp order, terminal evidence, and synchronization tolerance from an empty cache.
Run scene-group, episode, source-lineage, and content-hash disjointness checks and verify every data-derived transform traces only to train.
Reconcile raw, reviewed, excluded, selected, and per-split counts plus durations, factor coverage, outcomes, validity, behavior-pass, and quality rates with explicit denominators.
Write the card and generate its tables from manifests; add representative successes, a hard case, an excluded case, limitations, rights and privacy status, intended use, and unsupported claims.
Repair the highest-impact gate failure, compute final checksums, rerun the complete week workflow without modifying files, and preserve the verification result and reproduction command.
Promote the candidate to v1 only after every gate passes; otherwise issue a new candidate, then write the next concrete collection or evaluation experiment and the successor-version rule.

Observe

Documentation drift, missing media, split leakage, stale metrics, and quiet mutations become visible when one clean verifier regenerates the card facts from immutable manifests and checksums.

Done when

A fresh environment loads and inspects the selected episodes, all counts and hashes reproduce, split ancestry is disjoint, limitations match coverage, publication scope is authorized, and no v1 file must be edited to explain a failure.

Build today

Collect, review, and version real demonstrations for one useful tabletop or mobile task.

Evidence to save

DONE when the weekly ship note explains how “Freeze dataset v1 with card, examples, and quality metrics” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Calling a folder v1 as soon as the target episode count appears.

Better

Require finalized and loadable storage, exact manifests, reconciled metrics, split checks, provenance, card, rights review, representative examples, checksums, and a clean reproduction run.

Wrong

Editing a frozen episode label or split assignment in place after discovering an error.

Better

Preserve v1, publish a reviewed successor with changed hashes and metrics, and record the correction’s effect on training and evaluation claims.

Wrong

Writing limitations as vague caution text unrelated to data.

Better

Tie each limitation to missing or sparse conditions, operators, hardware, behaviors, or quality evidence and use it to bound intended claims and the next collection backlog.

Job connection

How this becomes employable evidence

Own a robot-dataset release gate that finalizes multimodal storage, verifies schema and lineage, freezes grouped splits and preprocessing provenance, generates denominator-backed cards and examples, and detects mutation through reproducible manifests and checksums.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 23 interview drill

Interview questions: Freeze dataset v1 with card, examples, and quality metrics

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

What makes a robot demonstration dataset genuinely frozen? Describe raw versus release lineage, finalization, manifests, grouped splits, checksums, card metrics, example selection, limitations, and the correction policy.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Does freezing dataset v1 mean the dataset is complete or high-performing?

Model interview answer

No. It means the exact contents, semantics, splits, evidence, checks, and limitations are immutable and reproducible; adequacy remains bounded by measured coverage and later evaluation.

Q2Why must LeRobotDataset v3 writers be finalized before checksums and release?

Model interview answer

Finalization flushes buffered metadata and closes Parquet or media writers; before that, files can be incomplete or unloadable and their fingerprints do not describe a valid final release.

Q3What should happen when a frozen label is proven wrong?

Model interview answer

Keep v1 unchanged and issue a traceable corrected successor with review evidence, new checksums and metrics, a change log, and the effect on prior claims.

Chapter starter artifact

Reject a scene group that leaks into held-out evaluation

Freeze Episode Foundry Dataset v1 with an immutable manifest, disjoint scene-group splits, synchronized episode records, raw failures, condition-cell coverage, a rights-aware dataset card, checksums, and a clean reproduction check.

week-23-dataset-v1-manifest.mjsJavaScriptDownload starter

const allowedSplits = new Set(["train", "validation", "heldout"]);
const cleanEpisodes = [
  { id: "e1", group: "scene-a", split: "train" },
  { id: "e2", group: "scene-b", split: "validation" },
  { id: "e3", group: "scene-c", split: "heldout" },
];
const plantedFailure = [...cleanEpisodes, { id: "e4", group: "scene-a", split: "heldout" }];
const missingGroup = [...cleanEpisodes, { id: "e4", split: "train" }];
const invalidSplit = [...cleanEpisodes, { id: "e4", group: "scene-d", split: "test" }];
const duplicateId = [...cleanEpisodes, { id: "e1", group: "scene-d", split: "train" }];

function reviewSplits(episodes) {
  if (!Array.isArray(episodes) || episodes.length === 0) return "REJECT manifest";
  const seenGroups = new Map();
  const seenIds = new Set();
  for (const episode of episodes) {
    if (typeof episode?.id !== "string" || !episode.id.trim()) return "REJECT missing_id";
    if (seenIds.has(episode.id)) return "REJECT duplicate_id";
    seenIds.add(episode.id);
    if (typeof episode.group !== "string" || !episode.group.trim()) return "REJECT missing_group";
    if (!allowedSplits.has(episode.split)) return "REJECT invalid_split";
    const previous = seenGroups.get(episode.group);
    if (previous && previous !== episode.split) return "REJECT " + episode.group;
    seenGroups.set(episode.group, episode.split);
  }
  return "PASS";
}

console.log("clean split: " + reviewSplits(cleanEpisodes));
console.log("planted group leak: " + reviewSplits(plantedFailure));
console.log("missing group: " + reviewSplits(missingGroup));
console.log("invalid split: " + reviewSplits(invalidSplit));
console.log("duplicate id: " + reviewSplits(duplicateId));
console.log("dataset v1: HOLD");

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-23-dataset-v1-manifest.mjs
Expected output: clean split: PASS planted group leak: REJECT scene-a missing group: REJECT missing_group invalid split: REJECT invalid_split duplicate id: REJECT duplicate_id dataset v1: HOLD
Planted failure to diagnose: The manifest includes a leaked scene group plus missing-group, invalid-split, and duplicate-ID cases; each malformed episode fails before Dataset v1 can freeze.