Chapter 16 · Operate a two-robot fleet across edge, cloud, and operator boundaries
Today in the field story
One problem, then the next
Mission M-204 needs a lifecycle from requested through accepted, assigned, executing, canceled, failed, or completed. Allocate work by capability, battery reserve, dock access, and the narrow aisle, then plant opposing reservations that deadlock the couriers. Apply a deterministic escape rule and preserve rejected assignments. The scheduler coordinates business intent; each robot still owns local feasibility and safe execution.
- Why now
The architecture needs a concrete mission and shared-resource model before interfaces can preserve it.
- Ignore today
Ignore globally optimal scheduling; prove feasible allocation, cancellation, and deadlock escape.
- Unlocks next
A versioned state machine that every API and event can reference.
Understand
Build the physical picture first
A fleet mission is a guarded state machine plus a shared-space promise: allocation chooses a feasible robot, traffic grants bounded access, and every interruption ends in an explicit recoverable state.
A mission is not one boolean called running. Use states that distinguish request validation, acceptance, queuing, assignment, dispatch, robot execution, blocking, cancellation, success, and failure. Define who may cause each transition and what evidence closes it. A cancel request is only an intention until the executor acknowledges it and measured behavior reaches the declared stopped or safely parked condition. Terminal states are immutable; a late running event must not resurrect a canceled mission.
Allocation begins with feasibility before optimization. Filter robots by capability, payload, map, operating mode, maintenance status, battery reserve, charger commitment, localization quality, and route availability. Only then compare cost such as travel time, expected finish time, or energy. A nearby robot at low state of charge can be an infeasible choice. Record the inputs and allocator version so a later incident can reproduce why the winning robot was selected.
Traffic coordination manages shared resources and planned movement, while robot-local sensing and protective behavior still handle immediate physical hazards. A corridor, lift, door, or charger can be represented by a reservation with owner, time or route scope, expiry, and release rule. Deadlock appears when participants hold resources while waiting cyclically for others. Detect the wait-for cycle or bounded no-progress condition, select a deterministic victim, retreat to a known holding point, and prevent an abandoned reservation from remaining forever.
Docking and charging are missions with preconditions, not magical battery side effects. The system needs a charger compatible with the robot, an approach route, an exclusive occupancy rule, dock detection, charge confirmation, timeout, retry bound, and a safe failure destination. Battery state is an estimate, so use a reserve and uncertainty margin instead of planning to zero. A disconnected robot retains its current local safety behavior while the mission becomes unknown or interrupted until evidence supports a new state.
Words you need
Name each idea precisely
- Mission lifecycle
The allowed states and guarded transitions from an external request through assignment, execution, cancellation, and exactly one terminal result.
Physical example:Delivery M-31 moves from accepted to queued, assigned to R2, running, and succeeded after arrival evidence; it cannot later return to running.
- Feasible set
The robots that satisfy every hard requirement for a mission before any cost-based ranking is applied.
Physical example:Only the robot with the correct tote interface, current localization, permitted map, and enough battery remains eligible.
- Reservation
A bounded claim granting one identified participant use of a shared spatial or operational resource under stated release and expiry rules.
Physical example:Robot R1 holds the narrow-aisle token until its rear footprint exits the aisle or the reservation times out into recovery.
- Deadlock
A no-progress condition in which participants wait on resources held by one another and none can complete without intervention.
Physical example:R1 holds the aisle entrance while waiting for a bay held by R2, and R2 waits for the same aisle to clear.
- State of charge
An estimate of remaining battery capacity, usually expressed as a percentage and paired with uncertainty and a required reserve.
Physical example:A robot at 25% may be rejected for an 8 m mission when the estimated 8% use would leave it below the 20% reserve.
- Holding point
A named, verified location where a robot can wait without occupying the contested resource or blocking planned traffic.
Physical example:The losing robot backs into bay H2 before the winning robot receives the aisle reservation.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalFleet arrival rates, backlog, and idempotency
A fleet service must keep up with robot events and process a retried mission exactly once at the business boundary.
- λ
- events arriving each secondUnit: events/s
- μ
- events safely processed each secondUnit: events/s
- B
- unprocessed backlog at one instantUnit: events
Two robots produce λ = 12 mission events/s while one consumer safely processes μ = 10 events/s.
Backlog grows at λ − μ = 2 events/s, so after 60 s the added backlog is B = 2×60 = 120 events.
Scale or slow admission before deadlines fail, and use one stable mission-event key so a retry updates the same business transition instead of repeating it.
It is familiar queue and idempotency engineering, but duplicated work can dispatch or cancel a physical mission rather than merely repeat a database write.
If λ = 8 events/s and μ = 11 events/s, does backlog grow under the stated steady rates?
No. Capacity exceeds arrivals by 3 events/s, so an existing backlog can shrink while those rates hold.
Using the stated battery cost per metre,
while
Only robot B preserves the declared reserve.
Allocate a delivery without violating battery reserve
Mission M-77 needs payload class tote on map L1. R1 and R2 both support the payload and map. R1 is 8 m away at 25% charge, R2 is 12 m away at 70%, estimated use is 1% per metre, and reserve is 20%.
Validate M-77 once, create its immutable identity, and move it from requested to accepted to queued; do not assign a robot before required fields and cancellation policy pass.
Build the feasible set from hard constraints: both robots satisfy payload, map, localization, and operating-mode requirements.
Estimate R1's finish charge as
25% - 8 × 1% = 17%; because 17% is below the 20% reserve, remove R1 from the feasible set.Estimate R2's finish charge as
70% - 12 × 1% = 58%; it preserves reserve and remains feasible.Assign M-77 to R2, record the battery inputs and allocator version, and request the route and any required aisle reservation before dispatch.
If R2 disconnects after assignment, mark execution unknown or interrupted, stop allocating conflicting work, and reconcile robot evidence before reassigning M-77.
Close the mission only from one terminal result correlated to M-77; a later duplicate running event is rejected by the state-transition guard.
The farther robot wins because it is the only battery-feasible candidate, and disconnect or duplicate events cannot create two active executions or resurrect the completed state.
Nearest-robot allocation is an optimization shortcut; production allocation first proves the robot can finish the entire mission within every hard constraint.
Physical examples
Where this appears in real life
Low-battery robot beside the pickup
Robot R1 is 8 m from a parcel at 25% charge; R2 is 12 m away at 70%. The route model estimates one percentage point of charge per metre and requires 20% reserve.
R1 would finish near 17% and is filtered out before distance ranking, while R2 remains feasible near 58% despite starting farther away.
Opposing robots at one aisle
Two floor tokens approach a one-token-wide paper aisle from opposite ends, each already occupying a small entrance zone and waiting for the other to move.
A wait-for cycle or no-progress timer chooses one victim, releases its claim, sends it to a named bay, and lets only the other robot enter.
Hands-on exercise
Make the idea observable
Use two simulated robots, one narrow shared lane, two holding points, one charger, and a small deterministic mission-state and allocation program.
Write the allowed mission transitions and guards, including queued cancellation, running cancellation, terminal immutability, disconnect, and evidence required for success.
Describe each robot with capability, map, pose, operating mode, payload state, battery estimate, reserve, charger compatibility, and last-state freshness.
Implement feasibility filtering before a deterministic cost function; log both rejected reasons and the winning cost inputs.
Represent the narrow lane and charger as reservations with owner, scope, acquisition time, expiry, release condition, and recovery action.
Run nominal delivery, incompatible payload, low battery, canceled docking, opposing-lane deadlock, and robot-disconnect cases from fixed inputs.
Restart the allocator during one running mission and verify durable mission state plus current robot evidence prevent duplicate dispatch or a second resource owner.
Hard constraints remove candidates before distance matters, and explicit reservations and transitions reveal deadlock, lost ownership, and cancellation races that a simple available flag hides.
Every scenario produces one allowed transition history, one allocator explanation, no overlapping shared-resource owner, a bounded deadlock escape, and exactly one immutable terminal outcome or explicit unresolved state.
Build today
Build a simulated two-robot FleetOps system with a mission API, WebSocket operator console, task allocation, fault injection, canary update, rollback, and acceptance report.
Evidence to save
DONE when “Mission workflows, task allocation, traffic, deadlocks, docking, and battery” runs from one documented command and the nominal plus boundary outputs are attached.
Common mistakes
Catch the wrong mental model
Assigning the closest robot before checking battery, capability, payload, or route constraints.
Construct the feasible set from all hard constraints first, then rank only eligible robots with a reproducible cost function.
Marking a mission canceled as soon as the API receives the cancel request.
Represent cancel-requested and canceling states, obtain executor acknowledgment, and confirm the declared physical or simulated stopped condition before terminal canceled.
Treating traffic coordination as a replacement for robot-local obstacle protection.
Use fleet coordination for planned shared-resource access while keeping validated local sensing, limits, and protective stop behavior independent.
Allowing a charger or aisle reservation to remain owned after a process restart.
Give claims an owner generation, expiry, durable mission link, and reconciliation rule so abandoned reservations fail visibly and recover within a bound.
Job connection
How this becomes employable evidence
Implement and review a fleet mission lifecycle, capability-and-energy allocator, shared-resource reservation model, charging workflow, and deterministic deadlock recovery while preserving robot-local protective behavior and a reproducible dispatch explanation.
Relevant target roles
- Robot Fleet Backend / Platform Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 16 interview drill
Interview questions: Mission workflows, task allocation, traffic, deadlocks, docking, and battery
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Given two robots, one narrow aisle, one charger, and uncertain battery estimates, design allocation and mission states. Explain feasibility versus cost, cancellation races, deadlock recovery, and what you do when the assigned robot disconnects.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why does R2 win the example even though it is four metres farther from the pickup?
R1 would finish at an estimated 17%, below the 20% reserve, so it is infeasible; R2 would finish near 58% and is the only eligible robot.
Q2What makes a cancel request different from a canceled terminal state?
The request expresses intent; terminal canceled requires the executor to accept or complete cancellation and evidence that execution reached the defined stopped or safely parked condition.
Q3How can a fleet break a shared-resource deadlock without improvising?
Detect a wait-for cycle or bounded no-progress state, choose a victim by a deterministic rule, release its claim, route it to a verified holding point, and retry within explicit limits.