Chapter 16

Operate a two-robot fleet across edge, cloud, and operator boundaries

Turn two simulated robots into a small but production-shaped fleet system whose authority, state, and evidence survive retries, delays, disconnects, overload, and bad releases. This chapter places work at the correct robot, edge, fleet, or cloud layer; models missions and shared traffic; preserves meaning across REST, WebSocket, and ROS 2 gateways; processes durable events under backpressure; builds a stale-aware operator HMI; rehearses canary rollback; and closes with a fault-injected acceptance run tied to logs, metrics, traces, and declared service objectives.

Before you start

Complete the ROS 2 graph, QoS, lifecycle, launch, simulation, navigation, behavior-tree, cancellation, and safe-recovery work from Weeks 5–12.
Bring forward two simulated mobile robots, a simple map with one shared narrow resource and two parking or charging locations, and one repeatable navigation mission that already succeeds from a documented command.
Read JSON, follow a state-transition table, calculate rates and percentages, and use a local Python or TypeScript service plus a browser UI without connecting the exercises to an unsupervised physical robot.
Keep robot-local collision handling, command limits, watchdogs, and stop behavior independent of the browser, fleet service, and internet; this chapter coordinates work but does not create a safety-rated control system.

By the end

Assign control, mission, coordination, history, update, and operator responsibilities to robot, edge, fleet, or cloud layers with explicit failure behavior at every boundary.
Implement a mission lifecycle and feasible two-robot allocator that accounts for capability, battery reserve, docking, shared-space coordination, cancellation, disconnects, and deadlock escape.
Design a versioned mission contract whose REST acceptance, WebSocket state stream, and ROS 2 action execution preserve identity, units, freshness, idempotency, cancellation, and one correlated terminal outcome.
Use PostgreSQL as durable mission history, Redis only for expiring coordination or live views, and keyed Kafka events with sequence checks, idempotent consumers, bounded queues, and visible lag.
Build an operator HMI that separates requested, accepted, and measured state; exposes stale video, transforms, and telemetry; and uses authorization, leases, local timeouts, and an independently verifiable stop path.
Package services and configuration as immutable release inputs, stage a canary, enforce health gates, and prove a compatible rollback of code, configuration, schema, and operating procedure.
Run a frozen fleet scenario matrix with named faults, correlate logs, metrics, traces, and robot evidence, and report every objective as pass or fail without hiding unsuccessful missions behind averages.

The field story

Run the narrow-aisle FleetOps shift

Courier-A and Courier-B share one narrow warehouse aisle, two docks, and a simulated battery constraint. A web operator submits mission M-204, loses the browser connection, retries, and later sees a healthy video beside stale mission state. Meanwhile one robot blocks the aisle and a canary release slows event processing. This is no longer a policy-training problem. The mission is to preserve authority, identity, ordering, cancellation, and observable failure across robot, edge, fleet, cloud, and operator boundaries.

Build FleetOps as a chain of explicit contracts. Mission M-204 receives one durable identity, task allocation respects capability and shared resources, REST acceptance maps to a revisioned WebSocket stream and a ROS 2 action, database state and outgoing events commit together, and the HMI distinguishes requested from measured reality. The canary must fail visibly and roll back as a compatible bundle. Week 17 will challenge this exact baseline through traceability, SIL, HIL, and customer acceptance, while Week 25 will revisit it as an operating service.

Why this chapter now: The course has working robot behaviors and guarded learned policies; production value now depends on coordinating missions and operators without moving safety-critical authority into an unreliable web or cloud path.
Ignore for now: Do not build a global fleet product, connect unsupervised hardware, or treat the browser as a safety controller. Qualify two simulated robots and one narrow resource.
This unlocks: The versioned FleetOps baseline becomes Week 17’s validation target and later Week 25’s deployment, telemetry, replay, and acceptance subject.
Proof you will leave with: Provide layer and failure-boundary tables, mission state machine, allocator cases, API schema, idempotency records, revisioned stream, ROS action correlation, durable/outbox evidence, lag and duplicate tests, HMI freshness and authority tests, canary manifest, rollback drill, traces, metrics, and SLO report.

Environment contractRepository-supported Node.js 22.13.0 or newer runs the event-order starter. The integrated mission uses ROS 2 Jazzy with Gazebo Harmonic and two simulated robots; local service, browser, and data components must be pinned by the project that assembles them.

Compatibility boundary: REST, WebSocket, DDS, database, cache, event-log, WebRTC, Open-RMF, container, and browser behavior can drift independently. Preserve versioned schemas and compatibility evidence rather than asserting untested cross-version support.
Smoke check: Run node week-16-audit-mission-events.mjs; then start the two-robot baseline, submit the same idempotency key twice, disconnect and resynchronize the HMI, complete one mission, and verify one terminal identity across API, event, ROS, and trace records.
Contract reviewed: 2026-07-25
Runtime evidence: The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
Drift risk: high

Today in the field story

One problem, then the next

Draw Courier-A, Courier-B, edge executors, fleet services, operator browser, and cloud history as owners with different deadlines. For every arrow, name identity, schema, clock, freshness, retry, authority, and disconnected behavior. Remove the browser and cloud links in simulation. Robot-local collision, command limits, and timeout behavior must remain effective while the fleet stops inventing work from stale state.

Why now: Reliable fleet behavior starts by placing responsibility at the layer that can meet its failure and timing contract.
Ignore today: Ignore vendor scale and multi-site topology; qualify one two-robot boundary map.
Unlocks next: An architecture whose failures can be injected without creating two authorities.

Understand

Build the physical picture first

A fleet is four cooperating control rooms with different clocks: the robot protects motion, the edge keeps local work alive, fleet services coordinate peers, and cloud services remember and distribute.

Place a responsibility where its worst acceptable delay and failure consequence can be met. Motor control, protective stops, and the final command watchdog stay close to sensors and actuators because their deadlines cannot depend on Wi-Fi or the internet. An edge computer may run perception, navigation supervision, a robot adapter, and bounded local recovery. A fleet service allocates missions and coordinates shared resources. Cloud services can hold history, analytics, user administration, and update artifacts, but losing the cloud must not remove the robot's ability to stop safely.

A boundary is more than a network arrow. Each robot-to-edge, edge-to-fleet, and fleet-to-cloud exchange needs an owner, authenticated peer identity, schema version, unit and frame convention, timestamp and clock source, freshness limit, retry rule, and disconnected behavior. Commands additionally need authority, expiry, acknowledgment, cancellation, and a terminal result. Telemetry can sometimes be sampled or coalesced; a stop request or mission transition cannot be silently treated as disposable display data.

Different rates expose bad placement. A 250 Hz motor loop has a 4 ms period, while a 20 Hz planner has 50 ms and a 1 Hz cloud heartbeat has one second. A 120 ms cloud round trip spans 30 motor cycles. It can be acceptable for uploading a summary, yet it is already too late for the motor loop. Latency averages are insufficient: choose the layer from a bounded deadline and define what happens when the bound is exceeded.

Disconnected operation needs a deliberate authority model. A robot that loses fleet contact may finish a locally safe segment, pause at a named holding point, or stop and wait, depending on risk and site policy. It must not invent new missions, continue forever from stale instructions, or accept commands from two controllers. Reconnection starts with identity and generation checks, a fresh authoritative snapshot, and reconciliation of unfinished work before ordinary event flow resumes.

Words you need

Name each idea precisely

Failure domain: A part of the system that can become unavailable or incorrect without automatically taking every other part down with it.
Physical example:
A cloud analytics outage removes remote charts while the robot-local watchdog and edge mission executor continue their declared safe behavior.
Edge computer: Compute located on or near the robot site that can make latency-sensitive operational decisions without a cloud round trip.
Physical example:
A cabinet computer hosts the fleet adapter and map service so two robots can report state even during an internet outage.
Control plane: The path that defines desired work, policy, configuration, or allocation rather than carrying every high-rate physical measurement.
Physical example:
The fleet service assigns mission M-204 to robot R2, while wheel odometry remains on the robot-local data path.
Freshness limit: The maximum permitted age of state or a command before the receiver must reject, degrade, or request a new snapshot.
Physical example:
A pose update older than two seconds is marked unavailable for dispatch even if its coordinates look numerically valid.
Authority lease: A time-bounded grant naming which controller may issue a class of commands, renewed only while its conditions remain valid.
Physical example:
The edge executor may command navigation for five seconds at a time; loss of renewal makes the robot reject later commands from that session.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalFleet arrival rates, backlog, and idempotency

A fleet service must keep up with robot events and process a retried mission exactly once at the business boundary.

λ: events arriving each secondUnit: events/s
μ: events safely processed each secondUnit: events/s
B: unprocessed backlog at one instantUnit: events

Two robots produce λ = 12 mission events/s while one consumer safely processes μ = 10 events/s.
Backlog grows at λ − μ = 2 events/s, so after 60 s the added backlog is B = 2×60 = 120 events.
Scale or slow admission before deadlines fail, and use one stable mission-event key so a retry updates the same business transition instead of repeating it.

Programmer analogy

It is familiar queue and idempotency engineering, but duplicated work can dispatch or cancel a physical mission rather than merely repeat a database write.

If λ = 8 events/s and μ = 11 events/s, does backlog grow under the stated steady rates?

No. Capacity exceeds arrivals by 3 events/s, so an existing backlog can shrink while those rates hold.

The three periods are

T_{\text{motor}}=\frac{1}{250}=4\ \mathrm{ms},\quad T_{\text{planner}}=\frac{1}{20}=50\ \mathrm{ms},\quad T_{\text{cloud}}=\frac{1}{1}=1\ \mathrm{s}.

A $120\ \mathrm{ms}$ delay spans $120/4=30$ motor cycles.

Place seven FleetOps responsibilities by deadline and outage behavior

A two-robot lab has motor control at 250 Hz, local planning at 20 Hz, mission allocation within two seconds, a 10 Hz robot-state feed, minute-scale history uploads, an operator console, and signed update artifacts.

Convert rates to periods: motor control gets 1 / 250 s = 4 ms, local planning gets 1 / 20 s = 50 ms, and robot state gets 1 / 10 s = 100 ms between expected samples.
Place motor control and the final motion watchdog on the robot because a 120 ms network excursion would consume 30 motor periods before a remote response returned.
Place planning supervision and the robot-facing adapter at the edge so an internet outage does not remove local navigation feedback, stop, or a bounded recovery decision.
Place cross-robot mission allocation and shared-resource coordination in the fleet service because both robots need one consistent owner for the aisle and charger decisions.
Place durable history, fleet-wide analytics, identity administration, and update storage in the cloud, while keeping cached approved artifacts and rollback inputs available at the site.
For every arrow, add identity, schema, timestamp, freshness, timeout, retry, and offline rules; add authority, expiry, acknowledgment, cancel, and terminal result to command arrows.
Remove each connection in turn and accept the architecture only when observed behavior matches the table without losing robot-local protective behavior or creating two command authorities.

Result

Fast and safety-relevant behavior remains robot-local, site operation survives a bounded cloud outage, fleet-wide decisions still have one owner, and every disconnection has an observable state instead of an implicit hope.

What this proves

Layer names do not make an architecture robust; deadline placement, single authority, and tested loss behavior at every contract do.

Physical examples

Where this appears in real life

Warehouse internet cable removed

Two simulated carts are moving between local stations when the site's internet link is removed, but the local network, edge computer, and robot processes remain powered.

Look for:

Robots retain local stop and watchdog behavior, the edge follows the predeclared offline mission rule, and cloud history shows a gap rather than pretending the robots are current.

Two supervisors reach one cart

A fleet service reconnects while an edge recovery session still believes it owns robot R1, so both attempt to send a new navigation command.

Look for:

Generation and lease checks allow one authority, reject the other command explicitly, and require state reconciliation before ordinary scheduling resumes.

Hands-on exercise

Make the idea observable

Use the two-robot simulator and four labelled local processes or cards named robot, edge, fleet, and cloud. Keep all commands inside simulation.

Inventory mission request, velocity command, pose, battery, map, video, health, audit record, configuration, and update artifact; record producer, consumer, rate, unit, and consequence of staleness.
Draw the four layers and place each item according to its deadline, required availability, and authority rather than according to which programming language currently implements it.
Create a boundary table with schema version, identity, timestamp source, freshness limit, timeout, retry budget, acknowledgment, and disconnected behavior for every connection.
Run one nominal simulated mission and attach a correlation ID that remains visible from API acceptance through robot-side terminal state.
Disconnect robot-to-edge, edge-to-fleet, and fleet-to-cloud links one at a time; record the exact detection time, state transition, allowed local behavior, and operator indication.
Reconnect each link with one deliberately stale command and one old generation number; verify both are rejected before a fresh snapshot restores service.

Observe

The same network failure has different consequences at different layers, and state that lacks freshness or authority metadata becomes unsafe to act on even when its payload parses correctly.

Done when

The architecture table predicts all three disconnection trials, only one controller owns each command class, stale or prior-generation commands are rejected, and cloud loss never removes the simulated robot's local stop path.

Build today

Build a simulated two-robot FleetOps system with a mission API, WebSocket operator console, task allocation, fault injection, canary update, rollback, and acceptance report.

Evidence to save

DONE when the learning log explains “Robot–edge–cloud architecture and failure boundaries” in five precise points and a checked example produces the predicted output.

Common mistakes

Catch the wrong mental model

Wrong

Putting every decision in the cloud because centralized services are easier to update.

Better

Keep decisions whose deadline or safe fallback cannot tolerate internet loss on the robot or site edge, and use cloud services for work whose outage consequence is explicitly acceptable.

Wrong

Drawing an arrow labelled telemetry or API without a failure contract.

Better

Specify identity, schema, units, clock, freshness, timeout, retry, authority, and disconnected behavior, then remove the link and compare observation with the declared result.

Wrong

Letting edge and fleet controllers both resume commands after reconnection.

Better

Use one time-bounded authority lease and a generation-aware reconciliation step so prior owners and stale sessions cannot issue new work.

Job connection

How this becomes employable evidence

Partition a commercial AMR stack across on-robot compute, a site gateway, fleet services, and cloud operations; specify versioned contracts and offline behavior; then reproduce link loss without creating stale operator state or remote dependence in the motion-safety path.

Relevant target roles

Robot Fleet Backend / Platform Engineer
Robot HMI / Control & Monitoring Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 16 interview drill

Interview questions: Robot–edge–cloud architecture and failure boundaries

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Draw a robot-edge-fleet-cloud architecture for two AMRs. Defend where control and mission responsibilities live, quantify one timing choice, and explain authority, freshness, and recovery when each network boundary disappears.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why must a 250 Hz motor loop not wait on a 120 ms cloud round trip?

Model interview answer

The loop has a 4 ms period, so 120 ms spans 30 cycles; the network delay is neither sufficiently small nor bounded for that control deadline.

Q2What additional fields turn a data arrow into an operational boundary contract?

Model interview answer

At minimum peer identity, schema, units and frames, timestamp and clock, freshness, timeout, retry, health, and disconnected behavior; commands also need authority, expiry, acknowledgment, cancellation, and terminal result.

Q3What should happen before scheduling resumes after a disconnected edge controller reconnects?

Model interview answer

Verify identity and generation, establish one current authority, obtain a fresh authoritative snapshot, and reconcile unfinished or terminal work before accepting ordinary deltas or new commands.

Chapter references

Open-RMF — Fleet Adapter TutorialMaintainer explanation of the fleet adapter boundary, task bidding, robot status and battery updates, navigation graphs, coordinate transforms, robot API callbacks, and REST-based fleet-manager integration.
ROS 2 Jazzy — Topics, Services, and ActionsOfficial distinction among continuous topics, short request-response services, and long-running actions with goal, feedback, cancellation, and result, plus subscription-side message-age and period statistics.
RFC 9110 — HTTP SemanticsPrimary HTTP method and status semantics, including the formal meaning of idempotence, why POST is not inherently idempotent, retry limits, resource creation, and the boundary of an HTTP response.
Apache Kafka 4.3 — Introduction and Core ConceptsOfficial event, key, topic, partition, producer, consumer, durability, retention, and ordering semantics, including the guarantee that order applies to records read from one topic-partition.
OpenTelemetry — SignalsProject definitions of traces as request paths, metrics as runtime measurements, logs as event records, and propagated context used to correlate a distributed fleet mission.