Chapter 16 · Operate a two-robot fleet across edge, cloud, and operator boundaries
Today in the field story
One problem, then the next
Expose M-204 through a versioned REST request with one stable idempotency key, map acceptance to an ordered WebSocket stream, and correlate execution to a ROS 2 action. Retry the timed-out create, race cancellation with success, and reconnect after deliberately skipping a revision. The client must reload an authoritative snapshot rather than guess, while unsupported schemas and expired commands stop before simulated motion.
- Why now
Three interfaces must preserve one business operation through retry, reconnect, and cancellation races.
- Ignore today
Ignore broad public API design; implement one mission contract end to end.
- Unlocks next
A correlated interface path with explicit terminal-state ownership.
Understand
Build the physical picture first
A mission crossing REST, WebSocket, and ROS 2 is one numbered parcel with three views: submission accepts it, streaming reports it, and robot execution alone can produce its terminal physical result.
Choose an interface from its interaction, not from fashion. HTTP REST endpoints fit resource creation, retrieval, and state-changing requests whose responses are bounded. A successful create response means the service accepted or created a mission resource; it does not prove that wheels moved or a parcel arrived. WebSocket carries an ongoing, bidirectional connection useful for operator snapshots, state changes, and acknowledgments. ROS 2 topics carry continuous streams, services suit short request-response operations, and actions suit long-running goals with feedback, cancellation, and a result.
Mission creation needs application-level idempotency because HTTP POST is not inherently idempotent. The client generates one stable idempotency key for one intended mission and reuses it only for retries of the identical canonical request. The server stores the key, request fingerprint, mission identity, and response atomically. A repeated matching request returns the same resource; a reused key with different content is a conflict. This prevents duplicate creation, but downstream consumers still need their own deduplication because delivery and processing retries remain possible.
A gateway preserves semantics, not only JSON field names. Define mission ID, robot ID, correlation and causation IDs, schema version, units, coordinate frame, acquisition timestamp, sequence or revision, command expiry, requested authority, and allowed state transitions. Translate a REST mission into a ROS 2 action goal with a stable correlation, map action feedback into versioned state events, and map cancel intent through the executor. Never convert an API timeout into mission failure unless execution evidence establishes that terminal result.
Reconnect and cancellation are race-prone. A WebSocket subscriber first obtains an authoritative snapshot with a revision, then applies only newer deltas in order; a gap triggers another snapshot instead of guesswork. Cancellation can race with success, dispatch, or disconnect, so the state machine decides which terminal event wins and records the losing event as rejected evidence. Commands carry expiry and authority so a delayed packet cannot become valid merely because its schema parses after reconnection.
Words you need
Name each idea precisely
- Idempotency key
A client-generated identifier reused for retries of one intended operation so the server can return the same resource instead of repeating the side effect.
Physical example:Three timed-out submissions bearing key K-77 all refer to mission M-204 rather than creating three robot trips.
- Request fingerprint
A canonical digest or comparable representation of the fields whose equality proves that a retried key still describes the original operation.
Physical example:Changing the destination while reusing K-77 produces a conflict because the stored and new mission bodies do not match.
- Correlation ID
A stable identifier propagated through components so records from one end-to-end operation can be joined without relying on timestamps alone.
Physical example:The API record, allocator decision, action goal, robot feedback, and UI event for one delivery all carry correlation M-204.
- Schema version
An explicit contract revision that lets a receiver validate fields and apply a compatible parser rather than infer meaning from payload shape.
Physical example:Gateway v2 rejects a v0 mission lacking metres and map-frame declarations instead of assuming defaults.
- Authoritative snapshot
A complete state view at a named revision from which a client can safely resume applying later incremental events.
Physical example:After a WebSocket gap, the console loads robot and mission revision 412 before accepting event 413.
- ROS 2 action
An interface for a long-running goal that can emit feedback, receive cancellation or preemption, and return a result.
Physical example:A navigation goal reports progress for twenty seconds, accepts a cancel request, and returns canceled only after the executor handles it.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalFleet arrival rates, backlog, and idempotency
A fleet service must keep up with robot events and process a retried mission exactly once at the business boundary.
- λ
- events arriving each secondUnit: events/s
- μ
- events safely processed each secondUnit: events/s
- B
- unprocessed backlog at one instantUnit: events
Two robots produce λ = 12 mission events/s while one consumer safely processes μ = 10 events/s.
Backlog grows at λ − μ = 2 events/s, so after 60 s the added backlog is B = 2×60 = 120 events.
Scale or slow admission before deadlines fail, and use one stable mission-event key so a retry updates the same business transition instead of repeating it.
It is familiar queue and idempotency engineering, but duplicated work can dispatch or cancel a physical mission rather than merely repeat a database write.
If λ = 8 events/s and μ = 11 events/s, does backlog grow under the stated steady rates?
No. Capacity exceeds arrivals by 3 events/s, so an existing backlog can shrink while those rates hold.
Without idempotency, three create attempts may yield
With one stable key , the required invariant is
across all retries.
Follow one mission through retries, execution, and cancellation
A client intends one delivery from A to B, uses idempotency key K-77, and times out after its first POST. The server committed mission M-204 before the response was lost.
Canonicalize the create request, calculate its fingerprint F-9, and atomically store
(K-77, F-9, M-204, accepted response)with the mission row.When the client retries K-77 with the same body, compare F-9, return M-204 and its current representation, and create no second mission.
Publish an accepted event with mission M-204, schema version, sequence 1, timestamp, and correlation ID; the allocator later emits assigned sequence 2.
Translate M-204 to one ROS 2 action goal whose feedback becomes newer mission events; keep API acceptance distinct from robot-side running and observed completion.
Receive a cancel request while feedback says running, move to cancel-requested, forward cancellation once, and continue showing measured robot state while the outcome is unresolved.
If the action returns canceled, write one terminal transition and publish its revision; if success had committed first, reject the later cancel terminal event according to the state rule.
Reconnect the browser from an old revision by loading the authoritative mission snapshot before applying subsequent WebSocket deltas.
Three transport attempts create one mission, API success never impersonates physical completion, cancellation produces one state-machine-approved terminal result, and the reconnected UI does not invent intermediate state.
End-to-end correctness comes from stable identity and guarded state transitions across every interface; no individual protocol supplies exactly-once mission meaning by itself.
Physical examples
Where this appears in real life
Courier ticket copied three times
A dispatcher submits delivery ticket K-77, loses the response twice, and presses retry with the same mission body before the third response arrives.
One durable mission identity is returned for all three attempts; changing the destination under K-77 is rejected rather than silently mutating or duplicating work.
Display reconnects after missing two updates
A browser last displayed revision 40, disconnects while revisions 41 and 42 occur, then first receives live delta 43 after reconnecting.
The browser does not apply 43 onto stale revision 40; it requests a current snapshot, verifies its revision, and only then resumes ordered deltas.
Hands-on exercise
Make the idea observable
Extend the local FleetOps lab with a versioned mission JSON schema, a small HTTP service, one ROS 2 action or equivalent existing simulated goal path, and a WebSocket operator client.
Define create, read, cancel, snapshot, event, action-goal, feedback, and terminal-result contracts with IDs, units, frames, versions, timestamps, sequence numbers, expiry, and validation errors.
Implement atomic idempotency storage for mission creation and reject the same key when its canonical request fingerprint differs.
Propagate one correlation ID from HTTP acceptance through allocation, robot goal, feedback, terminal state, and browser event; make each record searchable by it.
Force a response timeout before acknowledgment, retry before and after the mission commits, and verify all matching requests return one mission identity.
Cancel once while queued and once while running; inject a success-versus-cancel race and prove the transition table admits only one immutable terminal state.
Disconnect the WebSocket, omit an event, reconnect with a later delta, and verify the client requests an authoritative snapshot rather than applying a revision gap.
Send an unsupported schema, expired command, duplicate event, and prior-generation goal; capture each explicit rejection and confirm none reaches simulated motion.
Transport success, durable acceptance, robot execution, and physical outcome become visibly different milestones, while retries and reconnects expose where identity and revision checks are required.
One intended create yields one mission across all retry points, every interface preserves correlation and units, cancellation races end once, revision gaps trigger snapshots, and invalid or expired commands never execute.
Build today
Build a simulated two-robot FleetOps system with a mission API, WebSocket operator console, task allocation, fault injection, canary update, rollback, and acceptance report.
Evidence to save
DONE when a deterministic “REST, WebSocket, and DDS gateways: schemas, idempotency, and cancellation” failure test reports expected versus actual behavior and passes after the documented fix.
Common mistakes
Catch the wrong mental model
Treating HTTP 200 or 201 as proof that the robot completed the delivery.
Use the response only for the API operation it represents, then require correlated executor feedback, terminal state, and observed task evidence for mission completion.
Generating a new idempotency key on every retry.
Reuse one stable key for retries of the identical intended create, store it atomically with the mission, and reject reuse with a different request fingerprint.
Applying the first WebSocket delta after reconnecting to whatever state remains on screen.
Resume only when revisions are contiguous; otherwise load an authoritative snapshot and then apply strictly newer ordered deltas.
Mapping cancel-request received directly to robot stopped.
Model cancellation as a race-aware workflow and use executor acknowledgment plus measured or simulated stop evidence before declaring the canceled terminal state.
Job connection
How this becomes employable evidence
Design and test an idempotent mission API, a revisioned browser event stream, and a ROS 2 action gateway that preserve one business operation through retries, reconnects, cancellation races, command expiry, and robot-side terminal evidence.
Relevant target roles
- Robot HMI / Control & Monitoring Engineer
- Robot Fleet Backend / Platform Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 16 interview drill
Interview questions: REST, WebSocket, and DDS gateways: schemas, idempotency, and cancellation
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A client times out after creating a robot mission and retries while the browser reconnects mid-execution. Explain your idempotency record, schema and correlation fields, snapshot protocol, ROS 2 action mapping, and terminal-state race handling.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why does a stable idempotency key need a request fingerprint?
The key may be retried only for the same intended operation; the fingerprint detects accidental reuse with different mission content and prevents ambiguous mutation or duplication.
Q2Which ROS 2 interface best matches a long navigation operation that reports progress and supports cancellation?
A ROS 2 action, because it represents a long-running goal with feedback, cancellation or preemption, and a result.
Q3What should a WebSocket client do when its last revision is 40 and the next event is 43?
Treat the gap as unknown state, obtain an authoritative snapshot at a known revision, and only then resume applying later contiguous events.