Chapter 16 · Operate a two-robot fleet across edge, cloud, and operator boundaries
Today in the field story
One problem, then the next
Seal code, configuration, schema, maps, calibration assumptions, and procedures as one canary release. Deploy it first to Courier-A during an idle simulated window, inject event lag, and stop promotion when the health gate fails. Roll back image, configuration, schema-compatible state, and active mission procedure together. Courier-B remains the known-good comparison instead of receiving a floating latest tag.
- Why now
A working fleet still needs reversible change control before it can be operated responsibly.
- Ignore today
Ignore fleet-wide automation; rehearse one canary and one complete rollback bundle.
- Unlocks next
A versioned deployment baseline that Week 17 can validate and later operations can restore.
Understand
Build the physical picture first
A fleet release is a sealed travel kit, not a floating tag: code, image digest, configuration, schema, firmware assumptions, migration, health gates, and rollback procedure move together.
A container packages a process and its user-space dependencies, not the whole robot computer. Record CPU architecture, operating system and kernel expectations, device mappings, network ports, file permissions, storage, clocks, GPU or accelerator runtime, and ROS 2 middleware assumptions outside the image. Use an immutable image digest for the release identity because a mutable tag can point to different content later. Produce a manifest that binds each service digest to source revision, build evidence, and compatibility requirements.
Configuration is executable behavior and needs its own typed schema, version, validation, defaults, secret references, and audit trail. Reject an unknown or incomplete configuration before a robot enters service; do not silently borrow yesterday's values. Keep secrets out of images and ordinary config artifacts. A compatibility matrix states which application, configuration, database schema, firmware, map, calibration, and robot model combinations are supported, including whether each change can run beside the previous version during a staged rollout.
An OTA rollout starts only when the target identity, release authorization, artifact integrity, robot operating state, battery, connectivity, and rollback inputs pass. Stage the smallest useful canary group in a safe or idle operating window. Health gates include process startup, schema compatibility, heartbeat, mission acceptance, navigation or task smoke result, latency, crash rate, resource use, and unexpected stop behavior. A heartbeat alone proves neither useful work nor fleet compatibility.
Rollback is a designed transition, not the command to pull an older image. It may need backward-compatible database changes, restoration of configuration, firmware or driver compatibility, map and calibration selection, draining active missions, operator communication, and reconciliation of work created by the failed version. Destructive migrations that the previous version cannot read require a forward repair or a preplanned restore path; discovering this after canary failure is too late.
Canaries bound exposure, not statistical certainty. Updating two robots in a fleet of forty exposes 5%. If one fails, the observed canary failure rate is 50%, which is enough to stop a zero-failure gate but far too little data to estimate fleet reliability precisely. Halt expansion automatically, preserve logs and mission evidence, roll back the complete compatible bundle, and verify a known-good smoke mission before returning either canary to normal scheduling.
Words you need
Name each idea precisely
- Image digest
An immutable cryptographic identifier for exact container content, unlike a tag that may later be moved to another image.
Physical example:The release manifest pins
fleet-api@sha256:…, so both canaries pull the same verified bytes rather than whateverlatestmeans today.- Configuration schema
A versioned machine-checkable contract for configuration fields, types, required values, ranges, and compatibility.
Physical example:A robot with
speed_limit: fastfails validation because the release requires a bounded numeric metres-per-second value.- Canary
A small, deliberately selected first deployment group used to exercise health gates before wider exposure.
Physical example:Two idle simulated robots receive release 1.4 while the remaining thirty-eight continue on the known-good bundle.
- Health gate
A predeclared measurable condition that must pass before a rollout can continue to the next stage.
Physical example:Both canaries must start, load configuration, finish three smoke missions, remain below latency and crash thresholds, and show no unexpected stops.
- Rollback bundle
The complete known-compatible code, configuration, schema, firmware assumptions, data handling, and operator procedure needed to return service safely.
Physical example:Restoring the prior API image also selects its compatible config, reverses a safe additive flag, drains a new mission type, and reruns the old smoke test.
- Compatibility matrix
A tested table declaring which versions of interacting software, configuration, data, firmware, maps, and robot models can operate together.
Physical example:Fleet adapter 1.4 supports robot firmware 7.2 and config schema 3, while adapter 1.3 cannot read schema 4.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalFleet arrival rates, backlog, and idempotency
A fleet service must keep up with robot events and process a retried mission exactly once at the business boundary.
- λ
- events arriving each secondUnit: events/s
- μ
- events safely processed each secondUnit: events/s
- B
- unprocessed backlog at one instantUnit: events
Two robots produce λ = 12 mission events/s while one consumer safely processes μ = 10 events/s.
Backlog grows at λ − μ = 2 events/s, so after 60 s the added backlog is B = 2×60 = 120 events.
Scale or slow admission before deadlines fail, and use one stable mission-event key so a retry updates the same business transition instead of repeating it.
It is familiar queue and idempotency engineering, but duplicated work can dispatch or cancel a physical mission rather than merely repeat a database write.
If λ = 8 events/s and μ = 11 events/s, does backlog grow under the stated steady rates?
No. Capacity exceeds arrivals by 3 events/s, so an existing backlog can shrink while those rates hold.
Canary exposure is
One failure among two canaries gives
which fails the stated rollout gate.
Decide a two-of-forty canary after one failure
Release 1.4 targets a forty-robot simulated fleet. The plan sends it first to two idle robots and requires zero crash, configuration, or smoke-mission failures before expansion. One canary fails configuration validation.
Calculate canary exposure as
2 / 40 × 100% = 5%and record the exact two robot identities, hardware profiles, prior versions, image digests, and config revisions.Apply the declared gate: one failure among two canaries gives an observed
1 / 2 × 100% = 50%canary failure rate and violates the zero-failure rule.Stop the rollout before any of the remaining thirty-eight robots pull the release; do not reinterpret the gate because one canary happened to pass.
Preserve the failed validation input, service logs, release manifest, host facts, and correlation ID, then classify the cause without changing the deployed artifact in place.
Drain or cancel canary work according to the runbook and restore the previous image digest, configuration version, schema-compatible state, and operating assumptions.
Run the known-good startup and smoke-mission checks on both canaries and reconcile any mission created by release 1.4 before normal scheduling resumes.
State the evidence boundary: the canary decisively rejected this rollout gate but two samples cannot estimate the release's general fleet failure probability.
Only 5% of the fleet was exposed, expansion stopped on the first declared failure, evidence remained intact, and the complete known-good bundle passed before the canaries returned to service.
A canary's job is to enforce an exposure gate and make recovery possible, not to turn two successful or failed samples into a broad reliability claim.
Physical examples
Where this appears in real life
Update two carts before the night fleet
A warehouse has forty carts. Two idle units receive a pinned service image and config while the other thirty-eight remain on the last accepted release.
One failed smoke mission halts expansion, keeps the failure evidence, and restores the whole compatible bundle before either cart returns to scheduling.
Recipe label moved after approval
A kitchen approves a jar labelled current, but someone later replaces its contents while retaining the same label, so the next cook cannot reproduce the meal.
A digest acts like a tamper-evident content identity; a friendly tag remains useful for humans but is not the immutable release proof.
Hands-on exercise
Make the idea observable
Package the local FleetOps services and versioned lab configuration for two simulated canary robots; keep secrets in local secret references and never embed production credentials.
Build each service once, record its immutable digest, source revision, dependency evidence, target architecture, required ports, devices, storage, clocks, and middleware assumptions in a release manifest.
Define and validate a versioned configuration schema; test missing field, wrong unit, out-of-range value, unknown version, and absent secret reference before startup.
Write a compatibility matrix for service, database schema, configuration, robot simulator version, map, and calibration, including old-new coexistence and rollback direction.
Declare canary eligibility and gates for startup, configuration, heartbeat, three smoke missions, latency, crashes, resource use, and unexpected stop count.
Deploy the identical digest and config to two simulated robots, inject one incompatible configuration and one failed mission gate, and verify expansion halts automatically.
Execute the rollback runbook for service, configuration, schema state, and active missions; retain failed artifacts and audit events instead of overwriting them.
Rerun the known-good smoke suite, verify both canaries' release identities and mission histories, and produce a go, hold, or reject decision tied to every gate.
A container that starts can still be incompatible with host devices or mission semantics, while a tested rollback depends on configuration, data, and operating state as much as on image bytes.
The release is reproducible by digest, invalid configuration fails closed, canary expansion stops on either planted failure, rollback restores a compatible bundle, and both known-good smoke missions pass with intact audit history.
Build today
Build a simulated two-robot FleetOps system with a mission API, WebSocket operator console, task allocation, fault injection, canary update, rollback, and acceptance report.
Evidence to save
DONE when a 60–120 second uncut “Containers, versioned configuration, OTA canaries, and rollback” demo links to its command, logs or plots, result count, and honest failure note.
Common mistakes
Catch the wrong mental model
Deploying a mutable image tag and recording only the friendly release name.
Bind the release to immutable image digests and source/build evidence; keep tags as labels, not proof of the bytes that ran.
Calling a canary healthy when its containers stay up and send heartbeats.
Gate on configuration, compatible dependencies, representative mission outcomes, latency, crashes, resource behavior, and unexpected stops in addition to liveness.
Rolling back the application image while leaving a newer incompatible config or schema.
Test a rollback bundle covering code, configuration, data or migration state, firmware and driver assumptions, maps, calibration, active work, and operator procedure.
Using one of two canary failures to claim a precise 50% fleet failure probability.
Use the observation to fail the predeclared gate and stop exposure, while stating that two trials are insufficient for a precise population reliability estimate.
Job connection
How this becomes employable evidence
Create and operate a fleet release pipeline that pins exact container artifacts, validates configuration and host assumptions, stages only eligible canaries, halts on mission-level health gates, and restores a complete compatible runtime while preserving incident evidence.
Relevant target roles
- Robot Fleet Backend / Platform Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 16 interview drill
Interview questions: Containers, versioned configuration, OTA canaries, and rollback
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A two-robot canary has one failure in a forty-robot fleet. Explain your release identity, config and compatibility checks, automatic gate decision, rollback contents, mission reconciliation, and what the small sample can and cannot prove.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why is an image digest stronger release evidence than a tag?
The digest cryptographically identifies exact immutable content, while a tag can be moved or reused and therefore cannot by itself prove which bytes ran.
Q2What does one failure among two canaries in a forty-robot fleet establish?
It exposes 5% of the fleet, yields 50% observed failure within that two-robot sample, and fails a zero-failure gate; it does not precisely estimate fleet-wide reliability.
Q3What belongs in rollback besides an older container image?
Compatible configuration, schema or data handling, firmware and driver assumptions, maps and calibration, active-mission reconciliation, operator steps, verification, and the known-good release identity.