Chapter 25 · Operate the capstone as a reproducible, diagnosable robot service

Today in the field story

One problem, then the next

Release Train telemetry distinguishes reported state, age, source, and confidence from true physical condition. Mission APIs use stable idempotency keys, request fingerprints, legal transitions, cancellation, and reconciliation. Update campaigns bind signed metadata to compatible hardware and software, stage canaries, evaluate health gates, and preserve a tested prior version. A dashboard acknowledgment is not task completion, and a downloaded package is not an installed healthy release. The rollback path remains a first-class acceptance case.

Why now: Fleet operations amplify identity, ordering, freshness, and update mistakes across machines.
Ignore today: Do not implement a universal fleet protocol or perform unauthorized updates.
Unlocks next: Operational state and rollback contracts for multi-run acceptance.

Understand

Build the physical picture first

A fleet is a group of couriers carrying numbered job cards and signed update parcels across unreliable roads, while headquarters distinguishes requested plans from reported physical reality.

Fleet telemetry should describe observation, not wishful state. Send robot identity, boot and software identity, source timestamp, sequence, pose and frame, velocity, localization quality, battery, mode, active mission, fault set, and data freshness appropriate to the system. Separate desired state from reported state: an update service may desire release 25.4 while the robot still reports 25.3, and a mission service may request docking while sensors report no movement. Dashboards must show stale or unknown data rather than extending the last green value forever.

A mission API is a state machine around a physical promise. Give each client operation an idempotency key so a timeout and retry return the original operation instead of creating a second delivery. Validate legal transitions such as queued → assigned → executing → completed, and define cancellation, rejection, expiration, partial physical progress, and recovery. HTTP 202 Accepted proves only that the service accepted work for processing; completion requires correlated robot and task evidence plus an independently meaningful terminal condition.

Secure over-the-air updating is more than downloading through HTTPS. The device verifies authorized signed metadata, freshness, target identity, hardware and dependency compatibility, artifact digest, and policy before installation. Separate keys and roles according to an reviewed update architecture, protect device identity, and reject expired, wrong-target, or older unauthorized metadata. This lesson borrows Uptane security concepts but does not claim a small classroom updater is conformant or suitable for an industrial fleet.

Roll out in rings: lab, one canary, a small cohort, then broader fleet, with frozen health windows and automatic or operator-controlled holds. Monitor boot, process health, telemetry freshness, mission outcomes, resource and thermal behavior, and new fault rates against the same baseline. Rollback needs a known-good slot or image, compatible configuration and persistent state, and a safe mission recovery plan. If a migration or firmware step is irreversible, call it forward recovery, not rollback, and plan that boundary before deployment.

Words you need

Name each idea precisely

Telemetry envelope: A versioned report containing observation values plus robot, software, time, sequence, unit, frame, quality, and freshness context.
Physical example:
Robot R7 reports map pose (4.2 m, 1.1 m, 0.3 rad), source age 80 ms, sequence 912, localization quality degraded, and release digest d4….
Idempotency key: A client-chosen identity that makes repeated equivalent requests resolve to one logical operation and its existing result.
Physical example:
Two retried POST /missions calls with key dock-R7-104 return mission M104 rather than dispatching the robot twice.
Reported state: What the robot or independent observer currently measures and attests, kept separate from what fleet control requested.
Physical example:
The console desires charging, but the robot reports waiting_at_dock and zero charging current, so charging is not shown as complete.
Canary deployment: A deliberately small first rollout whose declared observation window and gates limit exposure before wider promotion.
Physical example:
One of twenty equivalent robots receives release 25.4 for ten frozen missions while nineteen remain on the known-good digest.
Rollback attack: An adversarial attempt to install an older, once-valid software or metadata version that may contain known weaknesses.
Physical example:
A captured signed package from last year is replayed, but trusted version and freshness metadata cause the robot to reject it.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalReliability, denominators, and fleet rates

Production robotics requires rates, raw counts, time windows, and recoverable releases.

availability: healthy service time divided by scheduled timeUnit: percent
MTTR: mean time to recoverUnit: minutes or hours
incidents/robot-hour: normalized incident rateUnit: 1/hour

Five robots each run 8 h, giving 40 robot-hours.
Two incidents occur, so rate = 2/40 = 0.05 incidents per robot-hour.
Keep raw counts and categories; compare versions under the same window before deciding whether reliability improved.

Programmer analogy

Use familiar observability and incident response, but join logs to robot ID, physical context, bag replay, and software/model version.

Three incidents across 60 robot-hours gives what rate?

3/60 = 0.05 incidents per robot-hour.

Canary exposure is

E=\frac{2}{40}\times100\%=5\%.

The unaffected population is

N_{\text{unaffected}}=40-2=38\ \mathrm{robots}.

Adjudicate one retried mission during a canary update

Fleet R has ten simulated robots. R7 is the only canary for release digest 25.4@d4…; a client creates dock request key dock-R7-104, loses its reply, and retries.

Validate the signed release metadata, artifact digest, target hardware class, configuration schema, model dependency, minimum firmware, version freshness, and known-good recovery artifact before R7 becomes eligible.
Install only on R7, boot into motion-inhibited health checks, and require reported digest d4…, resolved configuration hash, device manifest, telemetry freshness, and no blocking fault before mission work.
On the first API request, atomically bind idempotency key dock-R7-104 to mission M104; on retry, return M104 and its present state without allocating another mission.
Correlate API, allocator, adapter, robot, and dock telemetry through M104 while distinguishing desired docking from reported pose, motion, contact, charging current, and terminal state.
If R7 reports fresh telemetry but no charging current before the deadline, mark M104 failed with observed physical state, hold the rollout, and preserve the canary evidence instead of promoting on process health.
Exercise the preplanned recovery path, verify the reported prior digest, configuration and state compatibility, then run its recovery acceptance mission; label rollback complete only if all frozen recovery gates pass.

Result

The retry creates one mission, stale optimism never becomes completion, the canary failure stops fleet exposure, and recovery is judged by reported identity plus task evidence.

What this proves

Fleet correctness needs identity at both control planes: one logical mission across retries and one authorized software state across staged update and recovery.

Physical examples

Where this appears in real life

Duplicate elevator request

A delivery robot sends an elevator mission, the network drops the reply, and the client retries while the first request is already executing.

Look for:

One idempotency identity must return the original mission and physical progress; a new database row or 202 response could otherwise trigger two conflicting trips.

Signed parcel for the wrong machine

A cryptographically signed camera-firmware package is authentic but targets model C, while the receiving robot contains model B hardware.

Look for:

Signature validity is necessary but not sufficient; verify target hardware, dependency, version, freshness, policy, and recovery compatibility before installation.

Hands-on exercise

Make the idea observable

Use two simulated robot workers, a local mission service, synthetic telemetry, and signed-test fixtures with non-production keys. Do not update or command a real fleet.

Define a versioned telemetry schema with identity, source time, sequence, units, frame, quality, freshness, mission, desired release, reported release, and fault fields.
Implement a mission state machine and idempotent create operation, then inject a lost response and prove the retry returns the original mission without duplicate dispatch.
Make the operator view render fresh, stale, unknown, desired, and reported states distinctly; verify an old green packet becomes stale after its deadline.
Create test update metadata for correct target, wrong target, expired metadata, mismatched digest, and older version; accept only the authorized compatible fixture.
Roll the test artifact to one simulated canary, execute a frozen health window with one mission, inject a regression, and verify promotion stops.
Restore the known-good test artifact and validate boot, reported identity, configuration, persistent mission reconciliation, and one recovery task before recording the rollback outcome.

Observe

The hardest defects appear between planes: an API acknowledges work the robot never performs, desired software is mistaken for installed software, or process recovery hides incompatible mission state.

Done when

Retries cannot duplicate physical work, stale telemetry is visible, invalid update fixtures fail closed, canary regression blocks expansion, and recovery has a task-level result rather than a boot-only claim.

Build today

Build a SIL→HIL→hardware CI ladder with containers, diagnostics, fleet telemetry, rollback, failure replay, and a frozen 50-trial acceptance suite.

Evidence to save

DONE when the integrated “Fleet telemetry, mission APIs, secure OTA, and rollback” path is observable, cancelable, and leaves the prior baseline reproducible.

Common mistakes

Catch the wrong mental model

Wrong

Displaying the last telemetry packet as current until a replacement arrives.

Better

Evaluate source timestamp, receipt time, sequence, and freshness deadline; render stale or unknown explicitly and inhibit decisions whose required observation has expired.

Wrong

Assuming HTTPS plus a checksum makes an OTA system secure.

Better

Use an approved update design with trusted signed metadata, freshness and version checks, target compatibility, protected keys, artifact identity, staged rollout, and tested recovery.

Wrong

Treating 202 Accepted or a successful reboot as task completion.

Better

Follow the same identity to reported robot and physical terminal state, and verify recovery compatibility plus a declared mission outcome.

Job connection

How this becomes employable evidence

Build mission and telemetry contracts for intermittent networks, expose desired versus reported robot state to operators, stage authenticated compatible releases, stop on canary regression, and reconcile physical mission state through rollback.

Relevant target roles

Robot Fleet Backend / Platform Engineer
Robot HMI / Control & Monitoring Engineer
Robotics Deployment, Integration & Validation Engineer

Chapter 25 interview drill

Interview questions: Fleet telemetry, mission APIs, secure OTA, and rollback

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A client retries a mission while one robot is receiving an OTA canary. Design the identities, state transitions, telemetry freshness, update checks, rollout gates, and recovery evidence that prevent duplicate work and false completion.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What should a repeated mission request with the same idempotency key do?

Model interview answer

Return or reconcile the original logical mission and its current result without creating a second dispatch or repeating physical work.

Q2Why must desired and reported release versions be separate?

Model interview answer

The fleet can request an update that has not installed, booted, or passed health checks; only reported state shows the robot's observed software identity.

Q3When is an older-image boot not a completed rollback?

Model interview answer

When configuration, persistent state, firmware, devices, health, mission reconciliation, or the required recovery task has not been shown compatible.