Chapter 23 · Collect and freeze a task-shaped capstone dataset
Today in the field story
One problem, then the next
A simulation-first console enters the Episode Foundry with the Week 22 lease and limit contracts intact. The UI emits operator intent and renews a dead-man lease, while authenticated exclusive authority, freshness, limits, cancellation, and timeout live outside the browser. Visible latency is measured against accepted commands and simulated state, not button animation. Camera frames, state, intent, admission results, intervention, and terminal labels share an episode clock contract so later training does not guess which action followed which observation.
- Why now
Collection software is part of the data-generating system and can corrupt both safety and timing.
- Ignore today
Do not treat UI responsiveness as command acceptance or stopped motion.
- Unlocks next
A bounded, synchronized teleoperation and recording path.
Understand
Build the physical picture first
Safe remote collection is a spring-return handle connected through several locked gates: if the hand, browser, network, lease, or watchdog disappears, command authority expires locally.
Picture the complete path before drawing the Flutter or Web screen: an operator holds a control; the client emits short-lived intent; an authenticated gateway checks user, robot, mode, sequence, age, frame, units, and limits; an exclusive lease identifies the only current controller; an edge or robot process applies an even tighter bound and watchdog; the controller produces measured state. The UI is one participant, never the final safety boundary, and this teaching system is not a safety-rated control function.
Implement dead-man behavior as renewable authority, not as one long “start moving” command. Pointer or key release, touch cancel, focus loss, hidden page, route change, socket loss, permission change, or stale state stops renewal. Each command can carry episode_id, lease_id, robot_generation, seq, issued_monotonic_ns, expires_monotonic_ns, frame, units, and bounded action values. The receiver rejects expired, duplicate, out-of-order, wrong-generation, wrong-frame, or out-of-range messages before they reach motion code.
Cancellation is observable state, not a magic stop packet. A ROS 2 cancel request may be rejected or accepted, an executing goal may enter canceling, and a terminal canceled result may arrive before or after measured motion reaches the declared stop predicate. The collection console therefore shows request, gateway decision, executor state, watchdog expiry, commanded value, and fresh measured state separately. A physical emergency stop remains independent of Flutter, WebSocket, ROS cancellation, and the ordinary command watchdog.
Recording must join modalities without pretending they share one clock. Store raw operator intent, validated command, measured robot observation, camera acquisition identity, receive times, control events, task and variation IDs, and terminal outcome under one episode_id. Prefer acquisition timestamps on producer clocks, measure or record clock offset and uncertainty, preserve sequence gaps, and calculate pairing residuals. Display current command age, state age, video age, round-trip samples, dropped sequences, and recorder health; average latency alone hides tails that matter.
Use a two-phase episode close. First revoke authority and reach the measured safe or idle predicate; then stop producers, flush frames and metadata, finalize the episode, and run structural checks. A user pressing “save” must not truncate the evidence needed to prove that motion stopped. An incomplete writer, missing terminal frame, excessive synchronization residual, or unexplained sequence gap yields invalid_recording, while the raw files and events remain preserved for diagnosis.
Words you need
Name each idea precisely
- Dead-man renewal
A short-lived authority signal repeatedly renewed only while the operator, client, connection, permissions, and robot state remain valid.
Physical example:A held screen control renews a 250 ms simulated lease every 100 ms; lifting the finger stops renewal and the edge watchdog expires authority.
- Command expiry
The receiver-side time after which a command is rejected or no longer authorizes motion, even if it arrives later.
Physical example:A forward command delayed in Wi-Fi reaches the gateway after its
expires_monotonic_nsand is logged as stale instead of moving the rover.- Exclusive authority lease
A versioned grant that permits exactly one authenticated controller to issue bounded commands to one robot in one allowed mode.
Physical example:A second tablet may observe the arm but cannot command it while collector A owns the current lease generation.
- Pairing residual
The remaining time difference between an observation and the action or event paired to it after the declared clock-alignment method.
Physical example:A camera frame is paired with the nearest measured joint state at a residual of 7 ms, which is stored rather than hidden.
- Cancel state
The explicit progress of a cancellation from request through acceptance or rejection, canceling work, terminal result, and measured safe outcome.
Physical example:The UI shows
cancelingwhile simulated wheel velocity is still falling, then showscanceledonly alongside fresh zero-velocity evidence.- Robot generation
An identity that changes across restart or authority epochs so late messages from an older robot session can be rejected.
Physical example:After the simulator restarts as generation 42, a queued command stamped generation 41 cannot control the new process.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalDataset coverage and independent scenes
More frames from the same easy scene do not equal broader robot experience.
- N_scene
- independent physical setupsUnit: scenes
- coverage
- tested condition cells representedUnit: percent
- failure rate
- failed frozen trials divided by all trialsUnit: percent
Your matrix has 3 object positions × 2 lighting levels = 6 condition cells.
Data covers 4 cells, so coverage is 4/6 = 66.7%.
Collect the two missing cells before duplicating the easiest one; split by whole scene, not adjacent frames.
It is test-case coverage, not line coverage: the unit is a distinct physical condition.
Seven of ten condition cells are covered. What is coverage?
7/10 = 70%.
The update period is
Remaining lease time is
Verify one delayed-network dead-man and recording trial
A simulated rover accepts at most 0.20 m/s forward and 0.40 rad/s yaw. The client renews at 10 Hz, each command expires after 250 ms, measured state is expected every 50 ms, and the recorder pairs frames within a declared 20 ms residual.
Acquire authority for robot generation 8 and episode E157-04; send command sequence 101 with
vx=0.15 m/s, issue and expiry times, framebase_link, and units, then show intent, acceptance, and measured velocity on separate rows.Inject a 320 ms one-way delay for sequence 102. Because it arrives after the 250 ms expiry, the gateway logs
rejected_staleand the robot process never applies it.Allow sequence 103 to arrive, then hide the browser before the next 100 ms renewal. Confirm no visibility callback directly claims a stop; it only stops renewal while the edge watchdog owns expiry.
Record the lease-expired event and zero command, then inspect measured velocity samples until the frozen stop predicate
abs(vx) < 0.01 m/sis freshly true; retain the actual elapsed stop time.Issue a ROS 2 cancel for the episode action and keep cancel-requested, cancel-accepted, canceling, canceled result, and measured stop timestamps distinct in the trace.
Pair each camera frame to robot observation and applied command, reject pairs over 20 ms from the release manifest, and preserve their residuals and clock-alignment metadata in raw data.
Close the episode only after authority is revoked, measured stop is proven, writers flush and finalize, required terminal records exist, and a load-and-scrub check can reconstruct the complete trial.
The delayed command is rejected, focus loss ends renewal, the independent watchdog removes authority, measured state proves the simulated stop, cancellation remains inspectable, and the finalized recording preserves timing quality.
A responsive control surface becomes defensible only when every command is bounded and expiring, every loss path removes authority outside the UI, and every claimed outcome is synchronized with measured evidence.
Physical examples
Where this appears in real life
Finger slides outside the Flutter control
An operator holds a simulated gripper jog button, then the touch leaves the widget and the operating system delivers a pointer-cancel event instead of a normal pointer-up.
The app stops lease renewal on cancel, the external watchdog expires command authority, measured joint velocity reaches the declared stop bound, and the event remains in the episode trace.
Old rover command returns after reconnect
A WebSocket disconnects while a forward command is in flight; the simulator restarts, reconnects with a new generation, and the delayed message arrives afterward.
Expiry, sequence, lease, and generation checks reject the message; reconnect begins with a fresh authoritative snapshot and no control until authority is reacquired.
Hands-on exercise
Make the idea observable
Connect a minimal Flutter or Web client only to a harmless simulator. Expose a gateway, a robot-side or edge watchdog, ROS 2 action state if available, measured velocity or joint state, a synthetic or simulated camera stream, and local episode storage.
Draw the client→gateway→authority lease→edge watchdog→controller→measured-state path and assign validation, timing, logging, and stop ownership to each boundary.
Implement one held control with pointer or key down, release, cancel, blur, page-hidden, navigation, and disconnect handling; every loss path stops renewal without declaring the robot stopped.
Validate authentication, robot and mode, exclusive lease, generation, sequence, issue age, expiry, frame, units, and numeric limits at the gateway, then reapply conservative limits beside the simulator.
Show intent, accepted command, applied command, action or cancel state, measured motion, lease expiry, telemetry age, video age, recorder state, dropped sequence count, and latency percentiles separately.
Record one episode identity across raw intent, validation decisions, applied commands, measured state, camera frames, task and variation fields, clock metadata, interventions, and terminal outcome.
Add assertions, then inject release, pointer cancel, focus loss, socket loss, stale command, duplicate sequence, excessive value, prior generation, cancel rejection, frozen video, and recorder-write failure one at a time.
For every case, record expected state, observed state, timing, and the exact causal fix; verify simulator-side authority and measured stop, finalize only complete episodes, and retain invalid attempts with their reason.
Client events, network state, executor cancellation, measured stopping, and media recording fail independently, so a single green connection badge or stop button cannot represent collection safety or validity.
All injected loss paths remove simulated command authority outside the client, no stale or excessive command is applied, measured state confirms each stop, and a reopened episode reconstructs synchronized control and observation history.
Build today
Collect, review, and version real demonstrations for one useful tabletop or mobile task.
Evidence to save
DONE when a deterministic “Build a safe Flutter/Web teleoperation console with dead-man control, command limits, latency visibility, cancellation, and synchronized recording” failure test reports expected versus actual behavior and passes after the documented fix.
Common mistakes
Catch the wrong mental model
Treating pointerup or a Flutter button callback as the component that guarantees stopping.
Use client events only to stop renewal; enforce short command expiry and zero-or-safe behavior in an edge or robot process that remains effective when the client or network disappears.
Clamping velocity only in JavaScript or Dart before sending it.
Validate limits at every authority boundary and beside motion execution, reject non-finite, wrong-unit, wrong-frame, stale, or wrong-generation values, and log the decision.
Showing average round-trip latency as proof the path is timely.
Expose command and observation age, tail percentiles, jitter, dropped sequences, clock uncertainty, and the tested watchdog response because rare delay controls stale-command risk.
Ending recording at the instant cancel is requested.
Continue through cancel disposition, authority revocation, measured safe state, terminal observation, writer flush, and validation so the most important stopping evidence is not truncated.
Job connection
How this becomes employable evidence
Ship a Flutter or Web robot-collection console that integrates authenticated leased control, bounded and expiring commands, ROS 2 feedback and cancellation, independent watchdog behavior, visible latency and freshness, and episode-grade multimodal recording.
Relevant target roles
- Robot HMI / Control & Monitoring Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
Chapter 23 interview drill
Interview questions: Build a safe Flutter/Web teleoperation console with dead-man control, command limits, latency visibility, cancellation, and synchronized recording
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Design browser teleoperation and data capture when focus, touch events, WebSocket, camera, ROS action state, clocks, and the robot can fail separately. Locate the dead-man, limits, cancel state, and proof of stop.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Where must the decisive dead-man timeout be enforced?
In an edge or robot-side component near command execution, not only in Flutter, the browser, or a remote gateway that can disappear with the network.
Q2Why are `cancel accepted` and `robot stopped` separate facts?
Acceptance means the executor agreed to process cancellation; motion may still be decelerating or cancellation may fail internally, so fresh measured state must prove the stop predicate.
Q3What does a pairing residual tell a dataset reviewer?
It quantifies how far apart the paired observation and action remain after the declared clock-alignment method, allowing excessive or uncertain synchronization to be detected.