Chapter 04 · Production C++, Linux, and deterministic timing
Today in the field story
One problem, then the next
A recorded packet now crashes the Night-Shift Sensor Gateway only on the seventeenth event. Preserve that trigger, run the appropriate sanitizer, profile the measured deadline path, and store revision, configuration, order, timestamps, and terminal outcome with the fixture. Add the repaired case to the clean CI gate while stating what the exercised replay does not cover.
- Why now
The project needs a causal path from one failure through detector and repair to a repeatable regression.
- Ignore today
Ignore claims that a clean sanitizer or replay proves absence of all races and hardware faults.
- Unlocks next
A reviewable defect artifact and a gate that prevents the same exercised failure from returning.
Understand
Build the physical picture first
Debugging tools are different kinds of evidence: a sanitizer catches a rule violation, a profiler measures where time went, a replay recreates chosen inputs, and CI checks that the evidence still holds after a change.
C++ can compile code that later reads outside an array, uses an object after its lifetime, overflows a signed integer, or races on shared memory. Compiler sanitizers instrument a test build so some of these mistakes produce a precise report. AddressSanitizer targets memory errors such as out-of-bounds access and use-after-free. UndefinedBehaviorSanitizer checks selected invalid language operations. ThreadSanitizer detects many data races in instrumented code. They have runtime and memory cost, platform limitations, and different coverage, so run supported configurations in testing and do not assume one clean sanitizer run proves the absence of defects.
A profiler answers a different question: where did CPU time, allocations, blocking, or system calls occur during the measured run? Start with a concrete symptom and use the target-appropriate tool. A function appearing at the top of a CPU profile is not automatically a bug; it may perform necessary work. Compare against the deadline and call path, then change one cause and remeasure. For timing-sensitive robotics code, instrumentation itself can perturb results, so preserve the profiler configuration and confirm the final timing with a production-like build.
Deterministic replay records the inputs and context needed to exercise downstream behavior again: source timestamps, sequence numbers, payload bytes or parsed values, configuration, calibration identity, software revision, random seed where relevant, and terminal outcome. Replay cannot recreate electrical noise, kernel scheduling, bus contention, or a device fault unless those effects were captured or explicitly injected. It is nevertheless powerful because a failure that once depended on live timing can become a stable regression case for parsing, filtering, control decisions, and error handling.
Continuous integration is the repeatable gate that configures a clean build, compiles supported targets, runs tests and selected analysis, and preserves reports. It does not replace review, target hardware, long-duration timing, or system validation. Give every event a monotonic timestamp, sequence number, severity, subsystem, outcome, and correlation identifier so logs can be joined without guessing. Avoid logging secrets or unbounded raw sensor data. A useful failure record lets another engineer answer what input arrived, which code and configuration ran, what state was believed, what command was proposed, what safety rule changed it, and how the run ended.
Words you need
Name each idea precisely
- Sanitizer
Compiler-added test instrumentation that detects selected memory, undefined-behavior, or concurrency defects at runtime.
Physical example:AddressSanitizer identifies that packet parsing read one byte past a fixed buffer.
- Profiler
A measurement tool that attributes time, allocations, blocking, or other costs to code paths during a run.
Physical example:A profile shows image conversion consuming most of a camera callback's CPU budget.
- Deterministic replay
Re-execution of saved inputs and relevant context to reproduce downstream program behavior.
Physical example:A timestamped packet file reproduces the parser failure that occurred at command sequence 417.
- Structured event
A log record with named fields and consistent types rather than an unparseable sentence.
Physical example:A watchdog event stores monotonic time, command sequence, age in ms, limit, action, and run ID.
- CI gate
An automated clean-environment check that blocks a change when declared builds, tests, or analysis fail.
Physical example:A pull request cannot merge when the sanitizer replay job reproduces an out-of-bounds read.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalRates, deadlines, jitter, and memory budgets
Hardware-facing software must finish work predictably before its deadline.
- f
- loop frequencyUnit: hertz (Hz)
- T = 1/f
- time available for one loopUnit: seconds (s)
- jitter
- variation around the expected timingUnit: milliseconds (ms)
For a 100 Hz control loop, T = 1/100 s.
Convert: 0.01 s = 10 ms per cycle.
If work sometimes takes 13 ms, it misses the 10 ms deadline by 3 ms; measure the distribution, not only the average.
This is a strict rendering or audio-processing budget, except a missed robot deadline can destabilize an actuator.
How much time does a 50 Hz loop have per cycle?
1/50 s = 0.02 s = 20 ms.
For callback times ,
Because , one sample misses the deadline.
Turn one corrupted packet into a regression gate
A live run crashed after packet 417. The recorded packet declares a payload length of 12 bytes but contains only 8.
Preserve the exact packet bytes, source and monotonic timestamps, sequence 417, parser configuration, and software revision.
Run the parser test with AddressSanitizer and confirm the report points to a read beyond the captured buffer.
Add an explicit length check before accessing the payload and return a typed malformed-packet result.
Replay valid neighbouring packets and the malformed packet; verify valid outputs are unchanged and packet 417 is rejected.
Record the rejection as a structured event containing expected length, actual length, sequence, and outcome without dumping sensitive payloads.
Add the clean build, tests, and supported sanitizer run to CI and preserve the report for a deliberately failing pre-fix revision.
The captured trigger no longer crashes the parser, malformed input has an explicit outcome, and the defect returns as a failing automated gate if reintroduced.
The repair chain is strongest when trigger, detector, code change, behavior check, and clean-environment gate all point to the same defect.
Physical examples
Where this appears in real life
Black-box event recorder
After a rover stops unexpectedly, sequence-linked sensor, state, command, and watchdog records reconstruct the decision without relying on memory.
The recorder preserves causal fields and configuration while avoiding claims about physical effects it did not measure.
Slow station on an assembly line
Watching the final queue shows a delay, but timing each station reveals that image conversion—not motor command generation—uses the budget.
Profiling localizes measured cost; it does not by itself explain why the implementation is costly or whether the work is necessary.
Hands-on exercise
Make the idea observable
Use a toy packet parser in an isolated local project. Deliberately introduce an out-of-bounds read only in this disposable exercise and never run the broken binary near hardware.
Create one valid and one truncated byte vector with fixed sequence numbers.
Build the toy parser with debug symbols and the compiler's supported address sanitizer configuration.
Run the truncated input, save the sanitizer summary and non-zero result, and identify the invalid access.
Add a bounds check and an explicit parse outcome, then rerun both inputs under the same instrumentation.
Add structured JSON or key-value event output with run, sequence, expected length, actual length, and outcome.
Create a clean CI-style command that configures, builds, tests, and runs the replay from an empty build directory.
Write a limitation note naming one hardware or timing fault that this replay cannot reproduce.
The sanitizer explains a selected memory defect, while replay makes its trigger repeatable and the clean command makes the repair reviewable.
The pre-fix case fails with relevant evidence, the repaired valid and truncated cases pass with explicit outcomes, and one command reproduces the clean gate.
Build today
Create a Linux C++ sensor→filter→controller pipeline with device permissions, bounded timing, tests, CI, structured logs, and deterministic replay.
Evidence to save
DONE when a 60–120 second uncut “Sanitizers, profiling, deterministic replay, and CI” demo links to its command, logs or plots, result count, and honest failure note.
Common mistakes
Catch the wrong mental model
Treating a clean AddressSanitizer run as proof that the program has no memory or concurrency defects.
It detects selected defects only on exercised, instrumented paths; combine tools, tests, review, and target validation while stating coverage.
Optimizing the hottest function without connecting it to a requirement.
Relate the profile to an observed deadline, resource, or throughput limit, make one controlled change, and remeasure.
Saving only payload bytes and calling the run deterministic.
Also preserve order, source and monotonic timing, configuration, calibration identity, revision, seeds where relevant, and terminal outcome.
Job connection
How this becomes employable evidence
Convert a field crash or bad model input into a versioned replay scenario with sanitizer or profiler evidence, structured outcomes, and a CI regression gate.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robotics Application / ROS 2 Integration Engineer
- Robot Learning Deployment / Physical AI Integration Engineer
Chapter 04 interview drill
Interview questions: Sanitizers, profiling, deterministic replay, and CI
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A deterministic replay passes after a crash fix. Explain exactly what it proves, what live timing and hardware faults it cannot prove, and which sanitizer, profiling, CI, and target tests you would add.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What is the difference between a sanitizer and a profiler?
A sanitizer detects selected program-rule violations in instrumented execution; a profiler measures where resources such as time or allocations were spent.
Q2Why does replaying captured sensor values not prove the live sensor path is fixed?
Replay can exercise downstream behavior but does not recreate device electrical behavior, driver timing, permissions, bus contention, or uncaptured faults.
Q3What should a useful structured watchdog event contain?
At minimum a monotonic timestamp, run or correlation ID, command sequence, measured age, configured limit, resulting action, and outcome.