Chapter 25 · Operate the capstone as a reproducible, diagnosable robot service
Today in the field story
One problem, then the next
A failed Release Train mission receives one incident identity linking structured logs, diagnostics, metrics, ROS traces, core dump metadata, mission transitions, bags or input records, build identity, and operator observations. You reproduce the software path from recorded inputs where possible and state where timing or physical context differs. Sensitive artifacts receive access and retention boundaries. Replay that matches the symptom strengthens a hypothesis; it does not prove the original physical mechanism without the missing evidence.
- Why now
Diagnosability determines whether a field failure can become a controlled regression.
- Ignore today
Do not dump secrets, retain everything indefinitely, or call replay identical reality.
- Unlocks next
A fact-first incident bundle and repeatable regression input.
Understand
Build the physical picture first
Robot observability is an aircraft recorder assembled from gauges, event notes, timing traces, crash memory, and replayable inputs, all synchronized to one mission identity.
Diagnostics answer bounded health questions such as whether a camera is present, calibrated, fresh, and producing the expected shape. Logs record discrete events with fields a machine can query. Metrics summarize repeated values over time, such as pose age, inference latency, queue depth, temperature, and mission success count. A trace follows causal work across callbacks or services. None is the physical truth by itself: a healthy heartbeat can accompany frozen data, and a low average latency can hide one deadline-breaking tail.
Give every mission, action goal, robot, process, build, and configuration an identity that survives service boundaries. Use wall-clock time for human correlation and a monotonic clock for durations on one machine; record clock source and synchronization quality when comparing hosts. Sequence numbers and source timestamps help expose gaps and staleness. Log requested, accepted, commanded, measured, and terminal states separately so an investigator can see where a claim changed, rather than reading one optimistic completed line.
A core dump is a captured image of a crashed process's memory and execution context, useful with the matching executable and debug symbols. It can also contain tokens, object data, personal information, or proprietary model inputs, so collection, access, retention, redaction, and transfer need explicit policy. A stack trace points to where execution stopped, not necessarily why. Preserve the original core and symbols, but test crash capture on a harmless development process before depending on it during an incident.
Incident replay needs the exact executable, model, resolved configuration, calibration reference, recorded inputs, initial state, event order, time model, random seeds, and dependency identities. First reproduce the symptom, then change one hypothesis at a time. A replay that crashes at the same stack strengthens a software hypothesis; a replay that succeeds does not disprove cable noise, thermal drift, lighting, or physical contact that was never recorded. Mark facts, derived measurements, hypotheses, experiments, and conclusions as different evidence classes.
Words you need
Name each idea precisely
- Diagnostic
A current, bounded assessment of a component's condition, supporting observations, and severity rather than a generic alive flag.
Physical example:The camera diagnostic reports serial match, calibration hash, frame age 42 ms, expected 30 Hz, actual 29.8 Hz, and zero dropped frames.
- Metric
A numeric measurement aggregated or sampled over time with declared unit, labels, and collection window.
Physical example:A histogram shows inference latency in milliseconds and reveals a 240 ms tail that the 38 ms average concealed.
- Trace
Timestamped causal events that follow work through callbacks, queues, services, actions, or processes at finer detail than ordinary logs.
Physical example:Trace spans connect image receipt, preprocessing, policy inference, action submission, controller acknowledgment, and cancellation for mission M-204.
- Core dump
A protected snapshot of a crashed process's memory and execution state, interpreted with the matching binary, libraries, symbols, and platform.
Physical example:A segmentation fault leaves thread stacks and memory values that identify the failing function when opened with the exact release symbols.
- Incident replay
A controlled rerun of captured inputs and initial conditions against identified software to reproduce and test a specific failure hypothesis.
Physical example:Recorded images, joint states, mission events, and timing are fed to release 25.3 before and after one queue-ordering fix.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalReliability, denominators, and fleet rates
Production robotics requires rates, raw counts, time windows, and recoverable releases.
- availability
- healthy service time divided by scheduled timeUnit: percent
- MTTR
- mean time to recoverUnit: minutes or hours
- incidents/robot-hour
- normalized incident rateUnit: 1/hour
Five robots each run 8 h, giving 40 robot-hours.
Two incidents occur, so rate = 2/40 = 0.05 incidents per robot-hour.
Keep raw counts and categories; compare versions under the same window before deciding whether reliability improved.
Use familiar observability and incident response, but join logs to robot ID, physical context, bag replay, and software/model version.
Three incidents across 60 robot-hours gives what rate?
3/60 = 0.05 incidents per robot-hour.
Using correlated timestamps,
Reconstruct a crash after a late grasp cancellation
In a simulated or approved restricted setup, mission M-204 reports stale perception, cancellation starts, and the action client process crashes. The final physical or simulated tool state must be measured independently.
Freeze the incident window and copy logs, metrics, ROS 2 trace, core dump, recorded inputs, mission events, build and model identities, resolved configuration hash, symbols, clock metadata, and artifact checksums into read-only storage.
Build a fact timeline: pose age crossed 250 ms at monotonic 18.420 s, cancellation was requested at 18.553 s, process received
SIGSEGVat 18.561 s, and the independent tool-state recorder ended at its observed pose.State evidence limits: timestamps from another host have measured synchronization uncertainty; the core contains process memory but no contact-force measurement; the trace observes software scheduling but not cable or lighting conditions.
Open the core with the exact binary and symbols, identify the crashing thread and stack, then write at least two hypotheses such as a cancellation callback lifetime defect or earlier memory corruption.
Replay the preserved inputs and event timing on the original build without editing the bundle; record whether the symptom, stack, and terminal state repeat and how many attempts were made.
Change one candidate cause, rerun the same replay plus a non-incident regression case, and report
reproduced,not reproduced, orinconclusive; reserveroot cause confirmedfor evidence that excludes credible alternatives.
The bundle connects a stale-input response to a reproducible software crash while keeping cross-host timing uncertainty and unrecorded physical conditions visible.
Observability turns an incident into testable timelines and hypotheses, but only when identities align and missing sensors remain missing rather than being filled with confident stories.
Physical examples
Where this appears in real life
Dashboard green while the wheel is stuck
A rover process publishes heartbeats and accepts a mission, but tape wrapped around one wheel prevents travel while the controller's requested velocity remains nonzero.
Alive process, accepted command, encoder motion, independent displacement, motor current, and terminal mission are different observations; a green heartbeat covers only one.
Kitchen accident reconstruction
A timer, oven-temperature chart, recipe steps, security video, and broken-tray position each preserve different parts of a dropped-cake event.
No single artifact contains intent, order, timing, and physical cause; synchronize them, protect sensitive footage, and separate visible facts from guesses.
Hands-on exercise
Make the idea observable
Use a disposable local capstone simulation and synthetic, non-sensitive inputs. Deliberately crash only a harmless development process; do not crash a robot controller or production service.
Add structured fields for mission, robot, action goal, process, build, model, configuration, source timestamp, sequence number, and monotonic duration to one nominal path.
Create diagnostics for absent device, wrong identity, stale frames, low rate, and calibration mismatch, then verify each state changes independently of the process heartbeat.
Capture a trace around perception-to-action work and compare callback or queue timing with a latency metric; document tracing configuration and measured overhead.
Trigger a controlled development crash containing no credentials, verify the core or crash record can be found, and confirm the matching symbols identify a deliberate failure site.
Package logs, metrics, trace, crash record, inputs, manifest, and checksums under one incident identity with access and retention notes.
Replay once unchanged and once with a single candidate fix, then write a fact/hypothesis table that marks every unsupported physical conclusion as unknown.
Correlation gaps, clock ambiguity, missing symbols, overwritten logs, unrecorded inputs, and secrets inside crash artifacts often block diagnosis more than lack of log volume.
Another engineer can reconstruct the software timeline, locate the crash with matching artifacts, repeat the replay, and identify every physical fact that the bundle cannot establish.
Build today
Build a SIL→HIL→hardware CI ladder with containers, diagnostics, fleet telemetry, rollback, failure replay, and a frozen 50-trial acceptance suite.
Evidence to save
DONE when a deterministic “Diagnostics, tracing, core dumps, and incident replay” failure test reports expected versus actual behavior and passes after the documented fix.
Common mistakes
Catch the wrong mental model
Equating a heartbeat and normal CPU metric with a healthy sensor pipeline.
Check data identity, source timestamp, sequence continuity, rate, shape, calibration, and an independent task-relevant observation in addition to process liveness.
Sharing a core dump as if it were an ordinary text log.
Treat process memory as sensitive, apply approved collection, access, retention, and redaction controls, and keep the exact binary and symbols linked without exposing secrets.
Calling a matching replay stack trace complete proof of the field root cause.
Use replay to strengthen and test a software hypothesis, then account for unrecorded timing, hardware, environmental, and contact conditions before closing alternatives.
Job connection
How this becomes employable evidence
Instrument production ROS 2 and policy paths, correlate distributed incidents, preserve secure crash and replay bundles, distinguish scheduling or lifetime defects from missing physical evidence, and convert confirmed failures into regression cases.
Relevant target roles
- Robotics Software Engineer — ROS 2 / AMR
- Robotics Deployment, Integration & Validation Engineer
- Robot Learning Deployment / Physical AI Integration Engineer
Chapter 25 interview drill
Interview questions: Diagnostics, tracing, core dumps, and incident replay
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A robot mission ends after a node crashes during cancellation. Explain which diagnostics, logs, metrics, traces, core artifacts, clocks, and replay inputs you need, plus what each cannot prove.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why does average inference latency not establish deadline compliance?
The average can hide rare slow samples; inspect the declared distribution, tail, time window, load, and every deadline violation.
Q2What extra files make a core dump interpretable?
The exact executable, libraries, debug symbols, platform identity, and relevant source revision, handled under the same sensitive-data controls.
Q3What does a successful incident replay fail to prove?
It does not prove unrecorded physical conditions such as contact, cable noise, temperature, glare, wear, or cross-host timing occurred exactly as reconstructed.