Chapter 16 · Operate a two-robot fleet across edge, cloud, and operator boundaries

Today in the field story

One problem, then the next

Run the full narrow-aisle matrix with duplicate requests, lost stream revision, deadlock, stale video, cache loss, slow consumer, canceled action, and failed canary. Correlate API records, events, ROS feedback, logs, metrics, traces, and simulated robot outcomes under mission identity. Report success, latency, lag, recovery, and rollback objectives separately. This frozen FleetOps baseline now passes to Week 17 as a system to challenge, not a demo to admire.

Why now: Only an end-to-end fault matrix can show whether the distributed contracts agree.
Ignore today: Ignore customer deployment claims; account for every simulated scenario and objective.
Unlocks next: The explicit Week 16 to Week 17 validation target and a later Week 25 operations baseline.

Understand

Build the physical picture first

Fleet acceptance is a fire drill with numbered evidence: fixed scenarios create faults, correlated signals show where time went, and predeclared objectives decide pass or fail before anyone sees the result.

Acceptance starts with a frozen population and observable outcome. Define mission types, initial conditions, robot and software versions, fault schedule, success criteria, latency boundary, state-freshness boundary, recovery-time boundary, unsafe-attempt definition, exclusions, and trial count before execution. A service-level indicator is the actual measured quantity; a service-level objective is its target over a stated population and window. Neither is a guarantee of perfect operation, and neither replaces safety requirements or site acceptance.

Logs, metrics, traces, and robot artifacts answer different questions. Structured logs preserve discrete events with timestamp, severity, component, event type, mission and robot identity, configuration, and useful fields. Metrics aggregate rates, counts, distributions, and current gauges such as mission success, p95 latency, consumer lag, stale robots, or rollback count. A distributed trace follows one mission through API, database, allocator, gateway, and robot adapter using propagated context. Simulator recordings, ROS bags, screenshots, and terminal outcome evidence connect software signals to the observed task.

Correlation is designed before the incident. Propagate mission correlation, robot identity, attempt number, release version, and causation through synchronous and asynchronous boundaries; do not join by nearby timestamps alone. Clocks still need synchronization and uncertainty because a trace cannot correct an inaccurate robot timestamp. Control metric cardinality by using bounded dimensions such as mission type or result category rather than raw mission IDs in every time series; keep individual identities in logs and traces where targeted investigation belongs.

Fault injection tests a hypothesis, detector, containment, recovery, and evidence chain. Name the fault and expected behavior first, introduce it at a known time, confirm the system detected the right symptom, verify bounded degraded behavior, remove the fault, and measure recovery. Useful fleet cases include robot disconnect, stale pose, allocator restart, database slowdown, consumer backlog, WebSocket gap, frozen video, blocked aisle, invalid configuration, and canary failure. A fault tool reporting injected proves only the stimulus occurred, not that recovery worked.

Aggregate numbers must retain their denominator and failures. If 196 of 200 missions complete within 120 seconds, the measured indicator is 98%; it fails a 99% objective even when mean latency is attractive. Classify the four misses, publish their evidence, and report tail latency and freshness rather than averages alone. Simulation acceptance proves behavior within modeled software and faults; it does not prove real traction, wireless interference, safety circuitry, people interaction, or customer-site acceptance.

Words you need

Name each idea precisely

Service-level indicator: A precisely measured quantity representing service behavior for a declared population and window.
Physical example:
The fraction of delivery missions reaching observed success within 120 seconds is one fleet SLI.
Service-level objective: A target or acceptable range for an SLI, defined before measurement with its population, window, and exclusions.
Physical example:
At least 99% of the frozen 200 delivery trials must finish within 120 seconds.
Distributed trace: A correlated path composed of timed spans showing one operation as it crosses processes and services.
Physical example:
Mission M-204 has spans for HTTP acceptance, database commit, allocation, gateway dispatch, and robot action execution.
Structured log: An event record with stable named fields that can be filtered and correlated without parsing prose.
Physical example:
A cancel rejection records mission ID, current revision, attempted transition, component, release, and reason code.
Fault injection: A controlled introduction of a named failure used to verify detection, containment, recovery, and evidence against a prior expectation.
Physical example:
The test blocks robot R2 status for eight seconds and measures when dispatch removes it from the feasible set and when it safely returns.
Tail latency: Latency near the slow end of a distribution, commonly reported as a percentile rather than hidden by the mean.
Physical example:
Most missions allocate in 100 ms, but p95 allocation latency reaches 1.8 s during database slowdown and breaches the objective.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalFleet arrival rates, backlog, and idempotency

A fleet service must keep up with robot events and process a retried mission exactly once at the business boundary.

λ: events arriving each secondUnit: events/s
μ: events safely processed each secondUnit: events/s
B: unprocessed backlog at one instantUnit: events

Two robots produce λ = 12 mission events/s while one consumer safely processes μ = 10 events/s.
Backlog grows at λ − μ = 2 events/s, so after 60 s the added backlog is B = 2×60 = 120 events.
Scale or slow admission before deadlines fail, and use one stable mission-event key so a retry updates the same business transition instead of repeating it.

Programmer analogy

It is familiar queue and idempotency engineering, but duplicated work can dispatch or cancel a physical mission rather than merely repeat a database write.

If λ = 8 events/s and μ = 11 events/s, does backlog grow under the stated steady rates?

No. Capacity exceeds arrivals by 3 events/s, so an existing backlog can shrink while those rates hold.

The measured objective is

S=\frac{196}{200}\times100\%=98\%.

Because $98\%<99\%$ , the target fails, and the remaining $200-196=4$ missions require explicit failure categories.

Judge a 99% within-120-second objective

A frozen acceptance run contains 200 comparable delivery missions. The objective requires at least 99% to reach observed success within 120 seconds. Exactly 196 meet that condition.

Confirm the denominator is the predeclared 200 attempted missions and that exclusions were frozen before the run rather than removed after failures.
Calculate the indicator as 196 / 200 × 100% = 98% and count 200 - 196 = 4 missions that miss the combined success-and-latency condition.
Compare 98% with the 99% objective and mark the objective failed; do not round the result upward or replace it with average mission time.
Use correlation IDs to open the trace, structured logs, release identity, robot evidence, and relevant metric window for each of the four misses.
Classify each miss by the earliest supported cause such as allocation delay, blocked resource, stale robot state, executor failure, or outcome after 120 seconds; retain unknown when evidence is insufficient.
Check separate freshness, recovery-time, unsafe-attempt, and tail-latency objectives because passing or failing one SLI cannot substitute for the others.
Publish the failed objective, all four categories and links, the exact simulated conditions, and the boundary that real site behavior remains unverified.

Result

The measured service level is 98%, so the 99% objective fails; all four misses remain visible and individually traceable rather than disappearing inside a favorable average.

What this proves

An acceptance report is credible when the denominator, threshold, failures, and evidence boundary remain fixed even when they produce an inconvenient result.

Physical examples

Where this appears in real life

Timed warehouse evacuation drill

A team declares which exits, alarms, response times, and headcounts must pass, then blocks one route and records detection, rerouting, completion, and every exception.

Look for:

The drill succeeds or fails against criteria written before the blocked route is chosen; an alarm sounding is only detection, not proof that everyone recovered.

Parcel receipt across five desks

One numbered parcel receives a time stamp at intake, allocation, handoff, transport, and delivery while a separate tally board counts all parcels and failures.

Look for:

The receipt chain explains one slow parcel, the tally summarizes the population, and the final signed delivery ties software timing to the observed outcome.

Hands-on exercise

Make the idea observable

Use the completed two-robot FleetOps simulator, versioned release manifest, operator console, event services, and a repeatable local scenario runner. Keep the run isolated from real robots and production services.

Freeze at least thirty scenarios spanning nominal delivery, battery and docking, shared traffic, cancellation, retry, reconnect, backpressure, HMI staleness, canary, and rollback; record seed, initial state, expected result, and objective link.
Declare mission success, p95 latency, state freshness, detection time, recovery time, unsafe-attempt count, duplicate terminal count, and rollback objectives with population, window, and exclusions.
Propagate mission, attempt, robot, and release identities through structured logs and trace context; expose bounded metrics for rates, distributions, lag, stale robots, failures, and recovery.
Inject at least five named faults: robot disconnect, blocked aisle, slow consumer, frozen video or missing WebSocket event, and failed canary health gate; timestamp each stimulus and expected response.
Run the frozen matrix without changing thresholds, preserve raw results and simulator or ROS evidence, and classify every non-pass without rerunning it out of the denominator.
For one slow or failed mission, reconstruct the path from API through storage, allocation, gateway, robot execution, HMI, and observed terminal result; state any clock uncertainty or missing span.
Publish a pass-or-fail table for every objective, a failure taxonomy, rollback outcome, exact versions, residual risks, and the explicit simulation-versus-site evidence boundary.

Observe

Metrics reveal the population symptom, traces locate time across components, logs expose discrete decisions, and simulator artifacts decide whether the robot task actually reached its terminal condition.

Done when

All frozen scenarios and planted faults remain in the report, each objective has a reproducible numerator and denominator, every miss links to raw evidence, rollback is observed, and no simulation claim is presented as field or safety acceptance.

Build today

Build a simulated two-robot FleetOps system with a mission API, WebSocket operator console, task allocation, fault injection, canary update, rollback, and acceptance report.

Evidence to save

DONE when the weekly ship note explains how “Fleet acceptance with logs, metrics, traces, fault injection, and SLOs” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Choosing the SLO threshold after inspecting the run.

Better

Freeze the SLI definition, population, window, exclusions, threshold, and trial conditions before execution, then report the resulting pass or fail unchanged.

Wrong

Calling a fault case passed because the injection tool confirms the fault was sent.

Better

Require evidence of the intended stimulus, correct detection, bounded containment, recovery, final mission state, and no prohibited side effect.

Wrong

Using average latency and service heartbeat to summarize fleet health.

Better

Report tail latency, freshness, success denominator, lag, recovery, and failure categories; a live process and a good mean can coexist with stale or failed missions.

Wrong

Putting raw mission IDs on every metric label for easy searching.

Better

Keep metric dimensions bounded and use mission identities in correlated logs and traces, avoiding unbounded time-series cardinality.

Wrong

Presenting simulated fault acceptance as proof of customer-site safety or reliability.

Better

State exactly which modeled software, versions, scenarios, and faults passed, then list real wireless, traction, circuitry, people, and site conditions that remain unverified.

Job connection

How this becomes employable evidence

Define and execute a fleet acceptance matrix, instrument cross-service mission context, inject communication and capacity faults, diagnose objective misses with logs, metrics, traces, and robot artifacts, and issue an evidence-bounded release or rejection decision.

Relevant target roles

Robot Fleet Backend / Platform Engineer
Robot HMI / Control & Monitoring Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 16 interview drill

Interview questions: Fleet acceptance with logs, metrics, traces, fault injection, and SLOs

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Your dashboard shows 98% of 200 missions met a 99% objective. Walk through the decision, the four failures, trace and log correlation, fault-injection proof, tail and freshness metrics, and what simulation still cannot establish.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Does 196 of 200 missions within 120 seconds pass a 99% objective?

Model interview answer

No. The measured indicator is 98%, leaving four misses, so the predeclared 99% objective fails.

Q2How do logs, metrics, and traces differ during one fleet investigation?

Model interview answer

Logs record discrete structured events, metrics summarize runtime behavior across a population, and traces follow one correlated operation through components; robot or simulator evidence establishes the observed task outcome.

Q3What must be shown beyond proof that a fault was injected?

Model interview answer

The expected fault reached the target, was detected correctly, produced bounded degraded behavior, recovered within the objective, ended in the right mission state, and caused no prohibited effect.

Chapter starter artifact

Reject duplicate and gapped mission events

A two-robot simulated shift completes a frozen mission matrix through one versioned API and correlated evidence path, survives retries, duplicate and missing events, browser disconnect, deadlock, overload, and a failed canary, then reports mission SLOs and every rollback result without losing durable state.

week-16-audit-mission-events.mjsJavaScriptDownload starter

const events = [
  { id: "e1", mission: "M-204", sequence: 1, state: "created" },
  { id: "e1", mission: "M-204", sequence: 1, state: "created" },
  { id: "e2", mission: "M-204", sequence: 2, state: "assigned" },
  { id: "e3", mission: "M-999", sequence: 3, state: "moving" },
  { id: "e4", mission: "M-204", sequence: 4, state: "completed" },
];
const missionId = "M-204";
const seen = new Set();
let revision = 0, state = "none", duplicate = 0;
let foreignMission = 0, gap = 0, accepted = 0;
for (const event of events) {
  if (event.mission !== missionId) { foreignMission += 1; continue; }
  if (seen.has(event.id)) {
    duplicate += 1;
    continue;
  }
  seen.add(event.id);
  if (event.sequence !== revision + 1) {
    gap += 1;
    continue;
  }
  revision = event.sequence;
  state = event.state;
  accepted += 1;
}
const output =
  "mission=" + missionId + " state=" + state +
  " accepted=" + accepted +
  " duplicate=" + duplicate +
  " foreignMission=" + foreignMission +
  " gap=" + gap;
const expected = "mission=M-204 state=assigned accepted=2 duplicate=1 foreignMission=1 gap=1";
if (output !== expected) throw new Error("event audit mismatch: " + output);
console.log(output);

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-16-audit-mission-events.mjs
Expected output: mission=M-204 state=assigned accepted=2 duplicate=1 foreignMission=1 gap=1
Planted failure to diagnose: Event e3 belongs to another mission and e4 skips this mission's sequence 3; accepting either would cross-contaminate state or declare completion across an unknown transition.