Chapter 25 · Operate the capstone as a reproducible, diagnosable robot service

Today in the field story

One problem, then the next

The release manifest couples Model v1 to application build, container digest, host image, driver and firmware versions, device identities, configuration schema and hash, calibration, secret references, and rollback compatibility. Containers improve reproducibility but do not contain the physical machine or every host dependency. Provisioning starts from a known image, validates configuration before service start, and rejects an incompatible device or stale calibration. Mutable tags and manually edited production files are removed from the train's identity.

Why now: Deployment evidence is useless if the executed bytes and surrounding system cannot be identified.
Ignore today: Do not claim a container captures firmware, devices, calibration, or secrets.
Unlocks next: An immutable deployable manifest and deterministic startup gate.

Understand

Build the physical picture first

Deployment is a labeled field kit: container, host, devices, firmware, configuration, calibration, secrets, and recovery instructions must all match before the robot may act.

A Docker image packages a filesystem and process environment, not the entire robot. It does not automatically capture the host kernel, GPU or camera driver, device firmware, udev rules, clock setup, network quality, calibration files, mounted secrets, or the mechanism itself. Image tags such as capstone:latest can move; a content digest such as sha256:… identifies bytes. Even a pinned image can be wrong for the target hardware, so compatibility remains a tested manifest claim rather than a property of Docker.

Provisioning makes a blank or known host reach a declared state. Split it into a reviewed base image, packages and drivers, users and permissions, device rules, services, storage and log policy, time synchronization, network policy, and health checks. Make the process idempotent: applying it twice should converge rather than duplicate services or corrupt state. Save secrets outside source, container layers, logs, and evidence bundles; record the secret version or reference, never its value.

Treat configuration as a typed interface. Give its schema a version, define units and ranges, reject unknown or missing safety-relevant keys, and calculate a hash of the fully resolved configuration after defaults and environment overlays. Keep calibration separate but linked because two robots can run identical code with different camera intrinsics or joint offsets. A startup report should state desired configuration, loaded configuration, rejected fields, device identities, and whether motion remains inhibited.

Reproducibility also needs change and recovery records. Build from reviewed source, pin dependencies where appropriate, produce a software bill of materials, scan and sign according to the organization's process, and promote one immutable artifact rather than rebuilding per environment. Before rollout, test backup format, database or configuration migrations, downgrade compatibility, and a known-good recovery image. Rollback means restored service and compatible state; merely starting an older container is not proof that missions, maps, models, or firmware remain usable.

Words you need

Name each idea precisely

Image digest: A content-derived identifier for one exact container image, unlike a human-friendly tag that may later point elsewhere.
Physical example:
Two edge computers pull capstone:stable, but only the recorded sha256:8a… digest proves they received the same image bytes.
Provisioning: The controlled process that brings a host's operating system, drivers, accounts, devices, services, policies, and storage to a declared state.
Physical example:
A new robot computer gains the approved GPU driver, camera rule, service account, clock source, log limits, and disabled-motion startup service.
Resolved configuration: The final validated values actually loaded after defaults, files, environment overlays, and runtime selection have been combined.
Physical example:
The base file says max_speed_mps: 0.4, the lab overlay says 0.2, and the startup manifest records the loaded value and hash as 0.2 m/s.
Device manifest: A versioned inventory joining logical device roles to expected model, serial, firmware, driver, port, and calibration identities.
Physical example:
Logical wrist_camera must resolve to serial C42 with firmware 3.1, driver 2.8, and calibration hash 41bd…, not whichever camera appears first.
Rollback compatibility: Evidence that an older approved release can read current persistent state and safely operate with the installed firmware, model, schema, and configuration.
Physical example:
Release 12 can start after release 13 only because the mission database migration is reversible and the controller firmware still accepts its protocol.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalReliability, denominators, and fleet rates

Production robotics requires rates, raw counts, time windows, and recoverable releases.

availability: healthy service time divided by scheduled timeUnit: percent
MTTR: mean time to recoverUnit: minutes or hours
incidents/robot-hour: normalized incident rateUnit: 1/hour

Five robots each run 8 h, giving 40 robot-hours.
Two incidents occur, so rate = 2/40 = 0.05 incidents per robot-hour.
Keep raw counts and categories; compare versions under the same window before deciding whether reliability improved.

Programmer analogy

Use familiar observability and incident response, but join logs to robot ID, physical context, bag replay, and software/model version.

Three incidents across 60 robot-hours gives what rate?

3/60 = 0.05 incidents per robot-hour.

Ignoring overhead, shared transfer time is

t=\frac{(2.4\ \mathrm{GB})(8\ \mathrm{Gbit/GB})(5)}{0.1\ \mathrm{Gbit/s}}=960\ \mathrm{s}=16\ \mathrm{min}.

Build a deployment identity for one edge robot

A capstone release uses a containerized ROS 2 service, an exported policy, one depth camera, one controller, a lab configuration overlay, and persistent mission state.

Record source commit, CI build identity, exact container digest, base-image digest, package lock or dependency manifest, model hash, and software bill of materials; keep the friendly release tag only as an alias.
Add host image, kernel, architecture, GPU runtime, driver, ROS 2 distribution, time service, and provisioning revision, then mark each compatibility claim as tested or unknown.
Map wrist_camera and arm_controller to model, serial, firmware, stable device path, expected data shape or protocol, and calibration hash; fail startup if an identity or required calibration differs.
Validate configuration schema 3, merge base and lab overlays, normalize the result, hash it, and print units, important limits, and source layers without printing any secret values.
Run a no-motion preflight that checks devices, clocks, storage, logs, model load, action server lifecycle, and fallback availability before producing a readiness result.
Test recovery on disposable state: upgrade, write a representative mission record, restore or migrate as designed, boot the approved prior release, and verify data plus no-motion health before calling rollback viable.

Result

The release is represented by a joined manifest rather than capstone:latest, and any missing host, device, calibration, configuration, or recovery fact blocks the corresponding readiness claim.

What this proves

Reproducibility means another controlled target can reconstruct the same relevant state and detect mismatches before motion, not merely launch the same container command.

Physical examples

Where this appears in real life

Identical lunchboxes, different tools

Two sealed lunchboxes have the same outside label, but one contains a spoon and the other a fork; the dining table also differs in height.

Look for:

The container resembles the lunchbox, while host drivers, attached devices, calibration, and environment resemble the missing tool and table; matching one package does not match the system.

Camera order changes after reboot

A robot has wrist and room cameras. After a USB reconnect, enumeration order swaps, and software using /dev/video0 silently reads the room instead of the wrist.

Look for:

Provision stable identity from serial or approved device attributes, validate the expected stream and calibration, and keep motion inhibited on mismatch.

Hands-on exercise

Make the idea observable

Use a local development machine, the capstone container definition, mock device records, and disposable data. Do not provision, reboot, update, or change a real robot host.

Build the development image twice from the same reviewed inputs and record both resulting digests; investigate any difference rather than declaring bitwise reproducibility automatically.
Replace floating base and dependency references with reviewed version or digest policies where appropriate, while documenting how security updates will intentionally advance those pins.
Write a machine-readable deployment manifest joining image, model, host, driver, firmware, device, configuration, calibration, and persistent-schema identities.
Create a schema check that rejects a misspelled key, wrong unit, out-of-range limit, absent required value, and unknown device serial before the application can become active.
Run provisioning logic twice against a disposable target or dry-run representation and compare planned changes; the second pass should be empty or explicitly explain non-convergent work.
Rehearse rollback with disposable mission state, then record PASS, FAIL, or NOT VERIFIED separately for process start, configuration load, device readiness, state compatibility, and task execution.

Observe

Container identity is usually the easiest row; host drivers, device-to-calibration binding, resolved overlays, persistent-state downgrade, and secret handling create the larger reproducibility gaps.

Done when

One manifest reconstructs every relevant version, invalid configuration fails closed before motion, repeated provisioning converges, and rollback claims stop at the boundary actually tested.

Build today

Build a SIL→HIL→hardware CI ladder with containers, diagnostics, fleet telemetry, rollback, failure replay, and a frozen 50-trial acceptance suite.

Evidence to save

DONE when “Docker, provisioning, versioned configuration, and reproducible deployment” runs from one documented command and the nominal plus boundary outputs are attached.

Common mistakes

Catch the wrong mental model

Wrong

Calling robot:latest a reproducible release identifier.

Better

Record and promote the immutable image digest with source, dependency, model, and host manifests; use tags only as changeable human-facing aliases.

Wrong

Baking site configuration, calibration, and credentials into the image.

Better

Version and validate configuration and calibration as separately identifiable inputs, inject secrets through approved controls, and log references or hashes without values.

Wrong

Declaring rollback successful when the old process starts.

Better

Verify schema and data compatibility, device and firmware compatibility, health, command inhibition, and the required mission outcome under the rollback acceptance plan.

Job connection

How this becomes employable evidence

Package and provision an edge robot release, bind exact devices and calibration, validate layered configuration before activation, expose a reviewable manifest, and rehearse state-compatible recovery across a managed fleet.

Relevant target roles

Robot Fleet Backend / Platform Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 25 interview drill

Interview questions: Docker, provisioning, versioned configuration, and reproducible deployment

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A container works on one robot but fails after deployment to another. Show the host, device, firmware, configuration, calibration, secret, and persistent-state evidence you would inspect before blaming Docker.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why can two robots using the same image digest still behave differently?

Model interview answer

Their host kernel, drivers, firmware, devices, calibration, resolved configuration, clocks, network, persistent state, or mechanics can differ outside the image.

Q2What should happen when an unknown safety-relevant configuration key appears?

Model interview answer

Schema validation should reject startup or activation with a clear diagnostic; silently ignoring the key makes the loaded behavior ambiguous.

Q3What does an idempotent provisioner do on its second run?

Model interview answer

It converges on the same declared state without duplicating or corrupting resources, and reports any intentional remaining changes explicitly.