Chapter 16 · Operate a two-robot fleet across edge, cloud, and operator boundaries

Today in the field story

One problem, then the next

Store M-204 and its outgoing event intention in one durable transaction, rebuild expiring presence after cache loss, and process keyed events with sequence and identity checks. Inject the starter’s duplicate and gap, a consumer restart, a poison schema, and lag beyond the bounded queue. PostgreSQL remains mission truth, Redis remains disposable coordination, and event replay cannot apply the same durable transition twice.

Why now: Interface correctness collapses when storage and event delivery disagree under failure.
Ignore today: Ignore production cluster tuning; prove authority, outbox, replay, and backpressure semantics locally.
Unlocks next: Durable mission history and observable event processing for the operator and tests.

Understand

Build the physical picture first

Fleet data has a signed ledger, an erasable live board, and keyed conveyor lanes: PostgreSQL preserves decisions, Redis expires coordination views, and Kafka moves replayable events without erasing overload.

Give each store one declared authority. PostgreSQL is the durable system of record for mission identity, allowed transitions, idempotency records, assignment decisions, and audit metadata. Write a mission transition and its outgoing event intention in one database transaction, commonly through an outbox row, so a process crash cannot commit state while losing the event. Database constraints should reject duplicate identities and illegal revisions; a cache hit or a consumed message must not bypass those invariants.

Redis is useful for short-lived presence, expiring leases, rate limits, and derived live views because it is fast and supports time-to-live behavior. That convenience does not make it durable mission truth. A robot-presence key must include observed timestamp, generation, and expiry; missing or expired means unknown or offline, not stopped. Rebuild cache state from authoritative sources after loss, and use fencing or generation values so an old lease holder cannot continue after another owner acquires control.

Kafka stores events in partitioned logs. Records sharing a stable key such as mission ID are routed to the same partition when configured accordingly, and order is then read within that topic-partition; there is no automatic total order across every partition or external database write. Consumers can see duplicates after retries or restarts, so validate schema, use event identity and aggregate sequence, apply transitions idempotently, and commit consumer progress only after the durable effect succeeds.

Backpressure is a correctness condition because old robot state can look plausible. If 20 robots publish 10 events per second, input is 200 events per second. A consumer handling 150 accumulates 50 events each second and reaches 3,000 events of lag after one minute. Bound memory, expose event and time lag, set admission or degradation rules, and distinguish data classes: coalesce replaceable telemetry by robot, but never silently discard a mission terminal event, cancellation, safety signal, or audit transition.

Malformed and out-of-order events need explicit dispositions. Quarantine a poison payload with its raw identity and validation reason instead of retrying it forever at the head of a partition or silently skipping it. A sequence gap marks the aggregate incomplete and triggers replay or reconciliation. Dashboards should show input rate, processing rate, queue depth, oldest-event age, retry count, quarantine count, and last applied sequence so operators see staleness before it becomes a false live picture.

Words you need

Name each idea precisely

Transactional outbox: A durable row written in the same transaction as business state and later relayed to messaging, preventing committed state from losing its intended event.
Physical example:
Mission M-204 and outbox event assigned-3 commit together before a publisher sends the event to Kafka.
Time to live: A duration after which an ephemeral key expires and must no longer be treated as current evidence.
Physical example:
Robot R2 presence expires five seconds after its last validated heartbeat and the allocator removes it from the feasible set.
Partition key: The value used to route related events into one Kafka partition so their log order can be consumed together.
Physical example:
Every transition for mission M-204 uses M-204 as key and reaches the same topic-partition.
Consumer lag: The distance or age between the newest available event and the last event a consumer has durably processed.
Physical example:
A console projection is 3,000 records and 60 seconds behind even though its worker process still reports healthy.
Backpressure: The mechanism that prevents producers or queues from overwhelming bounded consumers by slowing, rejecting, sampling, or degrading according to data-class policy.
Physical example:
High-rate pose updates are coalesced to the newest state per robot while mission terminals remain lossless and ordered.
Idempotent consumer: A consumer whose repeated processing of the same event identity cannot repeat the durable business transition.
Physical example:
Replaying assigned-3 after a crash finds its applied event ID and does not assign a second robot.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalFleet arrival rates, backlog, and idempotency

A fleet service must keep up with robot events and process a retried mission exactly once at the business boundary.

λ: events arriving each secondUnit: events/s
μ: events safely processed each secondUnit: events/s
B: unprocessed backlog at one instantUnit: events

Two robots produce λ = 12 mission events/s while one consumer safely processes μ = 10 events/s.
Backlog grows at λ − μ = 2 events/s, so after 60 s the added backlog is B = 2×60 = 120 events.
Scale or slow admission before deadlines fail, and use one stable mission-event key so a retry updates the same business transition instead of repeating it.

Programmer analogy

It is familiar queue and idempotency engineering, but duplicated work can dispatch or cancel a physical mission rather than merely repeat a database write.

If λ = 8 events/s and μ = 11 events/s, does backlog grow under the stated steady rates?

No. Capacity exceeds arrivals by 3 events/s, so an existing backlog can shrink while those rates hold.

Input and backlog rates are

r_{\text{in}}=20(10)=200\ \mathrm{events/s},\qquad r_{\text{lag}}=200-150=50\ \mathrm{events/s}.

After $60\ \mathrm{s}$ ,

N_{\text{lag}}=50(60)=3{,}000\ \mathrm{events}.

Calculate overload and choose a bounded degradation policy

Twenty robots each publish ten telemetry events per second. One projection consumer sustains 150 events per second. Memory is bounded, mission events must not be lost, and replaceable pose events may be coalesced.

Calculate input as 20 × 10 = 200 events/s and service as 150 events/s, leaving a backlog growth rate of 200 - 150 = 50 events/s.
After 60 seconds, calculate 50 × 60 = 3,000 queued events; report oldest-event age as well as count because event sizes and partitions differ.
Classify mission transitions, cancellation, terminal results, and audit events as non-coalescible; classify intermediate pose display updates as replaceable by a newer validated pose for the same robot.
Key mission events by mission ID and pose events by robot ID, preserving sequence within each aggregate while acknowledging that different partitions have no single global order.
Apply a bounded policy: stop accepting optional analytics work, coalesce queued pose events by robot, preserve critical transitions, and alert before freshness breaches the operator threshold.
On consumer restart, replay from the last durably committed offset, deduplicate by event ID, reject stale aggregate sequences, and reconcile any detected sequence gap.
Accept recovery only when lag count and oldest age return below declared thresholds and the PostgreSQL mission history matches the event projection for every test identity.

Result

The overload remains visible and bounded, replaceable display traffic degrades deliberately, mission truth stays durable, and restart replay cannot repeat assignments or hide sequence gaps.

What this proves

An unbounded queue converts overload into delayed false confidence; a correct design makes capacity, freshness, and loss policy explicit for each class of fleet data.

Physical examples

Where this appears in real life

Shift log, magnetic board, and envelope lanes

A dispatcher writes signed mission transitions in a bound logbook, copies current robot locations to a magnetic board, and sends numbered envelopes down lanes chosen by mission ID.

Look for:

Erasing the board does not erase the ledger, envelopes from one lane stay ordered, and rebuilding the board requires replay plus freshness rather than memory.

Scanner produces faster than inspection

Twenty paper robots each place ten event cards per second onto a belt, while the reviewer removes only 150 of the 200 cards arriving each second.

Look for:

The visible pile grows by 50 per second; after 60 seconds it contains 3,000 delayed cards, so a running reviewer is not evidence of current state.

Hands-on exercise

Make the idea observable

Run local PostgreSQL, Redis, and Kafka-compatible services already available to the lab, using only generated two-robot events and no production credentials or live robot connection.

Create mission, mission-transition, processed-event, and outbox tables with unique mission revision and event identity constraints; commit one transition and outbox row atomically.
Relay outbox rows to a Kafka topic keyed by mission ID, then mark publication without deleting the durable audit identity needed for retry.
Store robot presence in Redis with timestamp, generation, and a short TTL; allow expiry to produce unknown or offline rather than measured stopped.
Consume mission events transactionally into a projection, record processed event IDs, and commit progress only after the PostgreSQL effect succeeds.
Inject a duplicate, reversed sequence, missing sequence, poison schema, consumer crash before offset commit, Redis flush, and a consumer slowed below producer rate.
Chart input rate, processing rate, record lag, oldest age, retries, quarantine count, cache expiry, and last applied aggregate sequence while each fault runs.
Recover through replay and authoritative reconciliation, then compare mission rows, outbox identities, projection revisions, and terminal counts for exact agreement.

Observe

A healthy process can serve stale state, cache loss is survivable only when Redis is not mission authority, and replay is safe only when consumers and database effects are idempotent.

Done when

All injected faults are visible, memory remains bounded, poison data cannot block silently, expired presence becomes unknown, duplicate replay causes no repeated transition, and durable history plus projection reconcile exactly.

Build today

Build a simulated two-robot FleetOps system with a mission API, WebSocket operator console, task allocation, fault injection, canary update, rollback, and acceptance report.

Evidence to save

DONE when the integrated “PostgreSQL, Redis, and Kafka event processing: ordering and backpressure” path is observable, cancelable, and leaves the prior baseline reproducible.

Common mistakes

Catch the wrong mental model

Wrong

Using Redis current-state keys as the durable source of mission truth.

Better

Keep allowed mission transitions and audit history in PostgreSQL, give Redis views TTL and generation metadata, and rebuild them after cache loss.

Wrong

Claiming Kafka preserves one global order for every robot and mission.

Better

State the actual boundary: order is consumed within a topic-partition, so choose stable aggregate keys and use revisions when events meet other partitions or stores.

Wrong

Committing the consumer offset before the database transition succeeds.

Better

Make the durable effect idempotent, commit it first, and advance progress afterward so a crash leads to safe replay rather than silent loss.

Wrong

Adding an unbounded queue when the consumer cannot keep up.

Better

Measure rate and age, bound capacity, shed or coalesce only declared replaceable data, protect critical events, and alert before freshness violates the operating threshold.

Job connection

How this becomes employable evidence

Own the mission system of record, transactional event publication, expiring robot-presence cache, partition strategy, replay-safe consumers, overload policy, and lag dashboards for a fleet whose messages can duplicate, reorder, stall, or fail validation.

Relevant target roles

Robot Fleet Backend / Platform Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Application / ROS 2 Integration Engineer

Chapter 16 interview drill

Interview questions: PostgreSQL, Redis, and Kafka event processing: ordering and backpressure

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A fleet projection falls behind while messages duplicate after a restart. Explain PostgreSQL, Redis, and Kafka ownership, your outbox and partition key, offset timing, sequence checks, backpressure policy, and proof that mission state remains correct.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1At 200 incoming and 150 processed events per second, what is the lag after one minute?

Model interview answer

Lag grows at 50 events per second, so after 60 seconds it reaches 3,000 events, with oldest age approaching one minute if arrival and processing remain steady.

Q2Why does a mission ID make a useful Kafka partition key?

Model interview answer

It routes transitions for that mission to one partition so consumers can read their log order together; it does not create total order across different missions or external stores.

Q3What should Redis presence expiry mean to the allocator?

Model interview answer

The robot's current state is unknown or offline and it should leave the feasible set; expiry is not evidence that measured velocity is zero or the robot safely stopped.