Chapter 16 · Operate a two-robot fleet across edge, cloud, and operator boundaries
Today in the field story
One problem, then the next
Build the operator view around requested, accepted, commanded, and measured state rather than one green status. Show independent ages for WebSocket data, video, transforms, and robot telemetry. Acquire one server-authorized lease for teleoperation, then lose browser focus and network renewal. A robot- or edge-side expiry must stop simulated commands while the UI visibly admits stale or unknown state.
- Why now
Operators need truthful state and bounded authority when media and command paths fail independently.
- Ignore today
Ignore polished styling and remote hardware; prove freshness, authorization, lease, and local timeout behavior.
- Unlocks next
An HMI whose evidence can participate in acceptance rather than mask failure.
Understand
Build the physical picture first
A robot HMI is a cockpit with independent instruments: operator intent, command acceptance, measured motion, map state, and video age must never collapse into one reassuring green icon.
An operational HMI separates what the user requested, what a service accepted, what the robot executor accepted, and what sensors physically or simulatively observed. A pressed forward control is intent, not velocity. An acknowledgment is receipt, not motion. A zero velocity measurement is stronger evidence but still needs timestamp, source, and freshness. Show mission revision, robot generation, connection state, last update age, command lease, operating mode, localization quality, and stop evidence so the operator can see disagreement instead of receiving a fabricated summary.
WebSocket state and WebRTC video are independent channels with different delay, loss, and clocks. A clear video can accompany a disconnected command path, and a frozen video frame can remain displayed while telemetry continues. Attach acquisition or capture time, receive time, sequence, and stream state; display age and visibly blank, watermark, or disable reliance on stale media. On a WebSocket revision gap, replace deltas with an authoritative snapshot before allowing actions that depend on current state.
Teleoperation authority must be bounded beyond the browser. Authenticate the operator, enforce role and site permissions server-side, verify the selected robot and mode, and acquire an exclusive authority lease. Commands carry sequence, generation, issue time, expiry, frame, units, and limits. The operator continuously holds a dead-man control, but the decisive stop is a robot- or edge-side timeout that ends motion when renewals cease. Browser focus loss, hidden tab, network loss, lease expiry, or authorization change stops renewal and produces visible degraded state.
Three-dimensional state is useful only with current transforms and geometry. Render the robot footprint, planned path, map frame, restricted zones, holding points, and obstacle or localization confidence with a shared timestamp policy. A pose transformed with an old map-to-odometry relation can draw a believable robot in the wrong aisle. The HMI should reject or mark stale transforms, state the coordinate frame, limit rendering rate independently of control rate, and never convert a graphical boundary into the only physical motion limit.
Timing arithmetic guides the lease but does not certify safety. At 20 commands per second the period is 50 ms. With 80 ms round-trip time and a 150 ms lease, only 70 ms remains after one round trip. Test jitter and one-way age, not only average RTT, and set the local watchdog from the validated stopping behavior and platform design. The exercise demonstrates stale-state and timeout logic in simulation; it does not make browser teleoperation safety-rated.
Words you need
Name each idea precisely
- Stale state
State whose age, sequence, generation, or clock uncertainty exceeds the declared limit for the decision being made.
Physical example:A pose received three seconds ago remains valid history but is too old to enable a live teleoperation command.
- Dead-man lease
A short command authority that remains valid only while an operator continuously signals intent and timely renewals reach a local watchdog.
Physical example:Releasing the control or losing browser focus stops lease renewal, and the robot-side timer commands the simulated base to stop.
- Measured state
Sensor- or estimator-derived evidence of what the robot is doing, kept distinct from requested and acknowledged commands.
Physical example:Measured wheel velocity remains nonzero after a stop request, so the console shows disagreement instead of declaring stopped.
- Role-based access control
Server-enforced permission that allows an authenticated role to perform only approved operations on approved resources.
Physical example:A viewer can inspect telemetry but receives a denied response when attempting to acquire teleoperation authority.
- WebRTC
A real-time media and data framework often used for low-latency audio or video, whose stream health and capture age remain separate from command state.
Physical example:The camera stream stays smooth while the WebSocket command channel disconnects, so the HMI disables commands despite visible video.
- Transform freshness
The age and validity of the coordinate transform used to place state or geometry in another frame.
Physical example:A fresh base pose combined with an old map-to-odometry transform is marked invalid for drawing the robot against restricted zones.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalFleet arrival rates, backlog, and idempotency
A fleet service must keep up with robot events and process a retried mission exactly once at the business boundary.
- λ
- events arriving each secondUnit: events/s
- μ
- events safely processed each secondUnit: events/s
- B
- unprocessed backlog at one instantUnit: events
Two robots produce λ = 12 mission events/s while one consumer safely processes μ = 10 events/s.
Backlog grows at λ − μ = 2 events/s, so after 60 s the added backlog is B = 2×60 = 120 events.
Scale or slow admission before deadlines fail, and use one stable mission-event key so a retry updates the same business transition instead of repeating it.
It is familiar queue and idempotency engineering, but duplicated work can dispatch or cancel a physical mission rather than merely repeat a database write.
If λ = 8 events/s and μ = 11 events/s, does backlog grow under the stated steady rates?
No. Capacity exceeds arrivals by 3 events/s, so an existing backlog can shrink while those rates hold.
The command period is
After an round trip, the remaining lease time is
Audit one 20 Hz teleoperation lease
The console sends bounded teleoperation renewals at 20 Hz, measured round-trip time is 80 ms for this sample, and the local authority lease expires after 150 ms without a valid renewal.
Calculate the nominal command period as
1 / 20 s = 0.05 s = 50 ms; every command carries robot generation, sequence, issue time, expiry, frame, and bounded velocity.Subtract the observed 80 ms round trip from the 150 ms lease to obtain 70 ms of remaining time; record that this sample does not bound one-way delay or jitter.
Acquire one exclusive lease after server-side authentication, role, robot mode, and site checks; reject a second operator rather than merging control streams.
Show requested velocity, gateway acceptance, executor acceptance, and fresh measured velocity on separate rows with their own timestamps and sources.
Remove browser focus so renewal stops; verify the robot- or edge-side timer expires authority and commands simulated zero without relying on a UI callback.
Keep a frozen video frame on screen during the trial and verify frame age, watermark, and disabled control make its stale status unmistakable.
Accept the test only when fresh measured velocity reaches the declared stop condition within its validated bound and the event trail explains request, expiry, command, and observation.
The operator can distinguish intent from effect, a local timeout stops simulated motion after renewal loss, and stale video or state cannot continue presenting itself as live command evidence.
Dead-man UI behavior matters, but the enforceable guarantee comes from exclusive leased authority, local expiry, bounded commands, and fresh measured stop evidence.
Physical examples
Where this appears in real life
Remote pallet cart with delayed stop evidence
An operator releases a held forward control; the API acknowledges the stop immediately, but simulated wheel velocity remains positive for another 180 ms.
The HMI shows requested stop, accepted stop, and still-moving measured state as separate facts until fresh velocity confirms the bounded stop.
Camera freezes while position keeps changing
A WebRTC test freezes one frame showing an empty aisle while WebSocket telemetry continues to report the simulated robot moving and obstacle state changing.
Frame age rises visibly, video is marked stale, teleoperation is disabled by policy, and current telemetry does not make the old image appear trustworthy.
Hands-on exercise
Make the idea observable
Build the operator console against the two-robot simulator with WebSocket state, a recorded or synthetic WebRTC-like video stream, a 2D or 3D map view, and no physical actuator connection.
Create separate HMI fields for intent, API acceptance, executor state, measured velocity, mission revision, robot generation, telemetry age, transform age, video age, lease owner, and lease expiry.
Require authentication and server-side roles, then implement exclusive authority acquisition and bounded commands containing sequence, issue time, expiry, frame, units, and robot generation.
Implement a held dead-man control whose renewals stop on release, focus loss, hidden page, disconnect, permission loss, or lease rejection; enforce expiry outside the browser.
Load an authoritative snapshot before deltas, detect a skipped revision, and prevent teleoperation while robot, transform, or command-channel state exceeds its freshness threshold.
Render footprint, path, holding points, and restricted zone in one declared map frame; inject an old transform and verify the view marks or removes misleading geometry.
Inject 300 ms network delay, dropped WebSocket, frozen video, duplicate sequence, prior robot generation, competing operator, and delayed measured stop.
For every fault, capture the HMI indication and robot-side or simulator evidence; fail the case if the screen merely disables a button without the local stop path executing.
Video, telemetry, command acknowledgment, transform, and measured motion can disagree independently, so freshness and source must be visible at the point of action.
All stale channels are unmistakable, unauthorized or prior-generation commands are rejected, only one authority lease exists, focus or network loss stops simulated motion locally, and fresh measured state confirms the result.
Build today
Build a simulated two-robot FleetOps system with a mission API, WebSocket operator console, task allocation, fault injection, canary update, rollback, and acceptance report.
Evidence to save
DONE when a comparison table for “Safe operator HMI and teleoperation with WebSocket, WebRTC, and 3D state” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Turning the robot icon green when the command API acknowledges a request.
Display intent, gateway acceptance, executor acceptance, and fresh measured state separately; only the last can support a claim about current motion.
Stopping motion only from a browser mouse-up or key-up handler.
Use those events to stop renewal, but enforce command expiry in a robot- or edge-side watchdog that survives focus loss, browser failure, and network interruption.
Leaving the last camera frame visible without age because the video element still renders.
Track capture and receive freshness, mark or blank stale media clearly, and disable operations whose visual evidence exceeds the declared limit.
Drawing a fresh pose through a stale transform and treating the result as current map truth.
Validate transform timestamp, generation, and frame chain; mark geometry unavailable when the composed state is not fresh enough for the action.
Job connection
How this becomes employable evidence
Build and validate a robot operations console whose state model, WebSocket recovery, video freshness, coordinate transforms, permissions, command leases, and independent timeout behavior let an operator act without mistaking intent or cached pixels for physical fact.
Relevant target roles
- Robot HMI / Control & Monitoring Engineer
- Robot Fleet Backend / Platform Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
Chapter 16 interview drill
Interview questions: Safe operator HMI and teleoperation with WebSocket, WebRTC, and 3D state
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Design remote teleoperation for an AMR when video, telemetry, and commands can fail separately. Explain requested versus measured state, freshness, reconnect snapshots, exclusive authority, dead-man expiry, and the evidence that motion actually stopped.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why is command accepted not the same as robot moving or stopped?
Acceptance proves a component received or admitted the request; fresh measured state from the simulator or robot is separate evidence of the resulting motion.
Q2At 20 Hz, 80 ms RTT, and a 150 ms lease, what do the basic numbers show?
Commands are nominally 50 ms apart and one measured round trip leaves 70 ms before expiry; the calculation still does not bound jitter, one-way age, or physical stopping.
Q3Which component must end motion when the browser disappears?
A validated robot- or edge-side watchdog must expire the authority or command lease and execute the bounded stop behavior independently of the browser.