Chapter 04 · Production C++, Linux, and deterministic timing
Today in the field story
One problem, then the next
Ship the Night-Shift Sensor Gateway as a flight recorder, not a single demo. Freeze its operating contract, run normal, malformed, stale, overload, saturation, sink-failure, cancellation, and replay scenarios, and preserve every result. The README must lead a clean reviewer through build, test, run, fault, metrics, and limitations while requested, limited, delivered, and observed states remain separately visible.
- Why now
The week’s engineering practices matter only when they survive one integrated acceptance path.
- Ignore today
Ignore real actuator acceptance; the fake sink proves only the declared software scenarios.
- Unlocks next
A trustworthy component baseline for the ROS 2 distributed system.
Understand
Build the physical picture first
The repository is a small flight recorder and test rig, not just a program: it must show what entered, what decision was made, whether the output stayed inside its limits, how the run stopped, and how another engineer can repeat it.
The week's deliverable is one complete sensor-to-controller pipeline built from the boundaries you designed. A source produces timestamped samples; validation rejects malformed or stale data; a filter produces a measured state; a controller compares that state with a setpoint; a limiter constrains the proposed command; a sink records or safely delivers it; and an event recorder preserves the outcome. Keep the first version one-dimensional and simulated. The hiring value comes from reliability evidence, not from pretending that a console program controls a full robot.
Define the operating contract before implementation. State sample unit and clock, nominal rate, maximum accepted age, control period, command range, queue capacity, overload policy, cancellation deadline, and terminal safe state. A watchdog should convert missing fresh input into an explicit bounded outcome rather than reusing an old command indefinitely. Every command record should distinguish requested, limited, delivered, and observed values. In the simulated build, delivered means accepted by the fake sink—not physical motion—and the documentation must say so.
Test the failure paths as deliberately as the normal path. Include a good replay, malformed sample, out-of-order sequence, stale timestamp, source disconnect, queue overflow, controller saturation, sink rejection, and stop request while work is waiting. Assertions should check outcomes and bounds, not sleep and hope. For timing, collect monotonic timestamps and report actual samples, deadline misses, command age, and shutdown time under a named load. A single successful run is a demonstration; repeated declared scenarios with artifacts are evidence.
A reviewable repository contains the target graph, checked-in build preset, source and tests, replay fixtures small enough to inspect, CI configuration, README, architecture and ownership note, timing report, sample structured events, and limitations. The README should lead a new engineer from prerequisites to clean configure, build, test, normal run, failure replay, and report interpretation. Do not hide failed trials. Close the chapter by explaining how RAII, value boundaries, Linux identity, bounded concurrency, instrumentation, and replay each remove a different class of uncertainty.
Words you need
Name each idea precisely
- Operating contract
The declared units, clocks, rates, freshness limits, capacities, deadlines, ranges, failure actions, and terminal state of a component or pipeline.
Physical example:Wheel samples are rad/s on a monotonic clock at 50 Hz and become stale after 60 ms.
- Watchdog
A mechanism that detects missing or late progress and triggers a predefined bounded response.
Physical example:If no fresh command arrives for 100 ms, the fake motor sink records a zero-command timeout state.
- Backpressure
A deliberate response when producers offer work faster than consumers can safely process it.
Physical example:A capacity-4 control queue rejects new configuration work instead of allowing unbounded command delay.
- Deterministic fixture
A fixed, versioned input and context bundle used to reproduce a particular program path.
Physical example:A CSV contains source time, sequence, measured speed, validity, setpoint, and expected terminal outcome.
- Acceptance criterion
A measurable pass or fail condition declared before evaluating the run.
Physical example:All commands remain within [-1,1], stale input reaches zero output within 100 ms, and shutdown completes within 200 ms.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalRates, deadlines, jitter, and memory budgets
Hardware-facing software must finish work predictably before its deadline.
- f
- loop frequencyUnit: hertz (Hz)
- T = 1/f
- time available for one loopUnit: seconds (s)
- jitter
- variation around the expected timingUnit: milliseconds (ms)
For a 100 Hz control loop, T = 1/100 s.
Convert: 0.01 s = 10 ms per cycle.
If work sometimes takes 13 ms, it misses the 10 ms deadline by 3 ms; measure the distribution, not only the average.
This is a strict rendering or audio-processing budget, except a missed robot deadline can destabilize an actuator.
How much time does a 50 Hz loop have per cycle?
1/50 s = 0.02 s = 20 ms.
Document the release invariant with its measured inputs:
The acceptance check compares behavior at with the documented overload behavior at .
Follow a stale sample to a safe terminal outcome
The pipeline runs every 20 ms. Its newest wheel-speed sample is 85 ms old, while the freshness limit is 60 ms and the previous delivered command was +0.35.
Read source and current monotonic timestamps in the same clock domain and compute age: 85 ms.
Compare age with the 60 ms contract and classify the input as stale before filtering or control.
Do not reuse +0.35; create a stale-input outcome carrying sequence, measured age, and configured limit.
Pass the predefined zero or inactive command through the same limiter and fake sink used by normal commands.
Record requested prior command, safety-modified command, sink acceptance, watchdog reason, and run ID as separate fields.
Measure elapsed time from freshness violation detection to the recorded safe terminal output.
Replay the fixture and assert that the stale outcome, bounded command, fields, and response deadline remain unchanged.
The pipeline converts an old observation into an explicit, replayable zero-output outcome instead of silently continuing with a stale command.
Failure handling belongs in the normal typed data path so it can be bounded, logged, replayed, and tested.
Physical examples
Where this appears in real life
Elevator controller record
The controller logs requested floor, sensed position, door interlock, limited motor command, and terminal state for each run.
A command request is kept separate from interlock approval and observed movement, just as the project separates proposed, limited, delivered, and observed state.
Kitchen order rail with four slots
When every slot is full, the kitchen must stop accepting orders, replace an older order by policy, or redirect work; stacking paper without limit only hides overload.
A bounded queue makes overload visible and forces a choice tied to freshness, completeness, and safety.
Hands-on exercise
Make the idea observable
Build and publish locally the simulated C++ repository; use a fake command sink or an unpowered indicator only, never an uncontrolled motor.
Write the operating contract and acceptance table before the final implementation run.
Implement timestamped source, validation, filter, controller, limiter, fake sink, watchdog, structured recorder, and bounded shutdown using narrow interfaces and RAII owners.
Create CMake targets for production libraries, the pipeline executable, and tests; add a shared preset and clean build instructions.
Add replay fixtures for nominal, malformed, out-of-order, stale, overflow, saturated, rejected-sink, and cancel-while-waiting cases.
Run supported sanitizers on tests, then collect at least 1,000 timing cycles under declared idle and background-load conditions.
Configure CI to run a clean build, tests, selected instrumentation, and deterministic replays while retaining concise reports.
Ask another engineer or use a new empty checkout-equivalent folder to follow the README, then fix every undocumented step they encounter.
The same production calculations serve live-simulated and replay paths, while each failure has a typed outcome, bounded command, timing record, and repeatable gate.
A clean environment follows one documented path to build and pass all declared scenarios; the report includes raw counts and tails, shutdown is bounded, failure artifacts remain visible, and limitations clearly state that fake-sink acceptance is not physical-actuation proof.
Build today
Create a Linux C++ sensor→filter→controller pipeline with device permissions, bounded timing, tests, CI, structured logs, and deterministic replay.
Evidence to save
DONE when the weekly ship note explains how “Ship the deterministic control-pipeline repository” changed the build, what still fails, and the first task for next week.
Project gate
Gate 1 · Explain transforms and PID without notes; ship a tested C++ control loop.
Common mistakes
Catch the wrong mental model
Calling the project complete when the normal demo prints correct commands once.
Predeclare and repeat normal, malformed, stale, overload, saturation, sink-failure, cancellation, and clean-build acceptance scenarios.
Logging one final command without its source age or safety changes.
Record correlated source, requested, limited, delivered, and observed states with timestamps and the reason for each modification.
Claiming real-time or hardware control from a desktop fake-sink run.
Report the measured platform and conditions precisely; fake-sink tests prove software behavior for those fixtures, not physical timing, drivers, or actuator response.
Job connection
How this becomes employable evidence
Present a repository that demonstrates the production C++ and Linux proof missing from the current portfolio: typed contracts, deterministic build, bounded runtime, observable failures, replay, and CI.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robotics Application / ROS 2 Integration Engineer
- Robot HMI / Control & Monitoring Engineer
- Robot Fleet Backend / Platform Engineer
Chapter 04 interview drill
Interview questions: Ship the deterministic control-pipeline repository
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Walk through your pipeline from timestamped sample to terminal command. Then explain ownership, queue policy, stale-data behavior, shutdown, measured timing, replay coverage, and the strongest claim the fake sink does not support.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why must the pipeline record requested and limited commands separately?
It shows what policy proposed, what the safety or range boundary allowed, and whether a limit or stale-data rule changed the command.
Q2What is the correct response when a bounded queue is full?
Apply the predeclared policy—reject, drop a chosen item, block for a bounded time, or enter a safe state—and record the outcome; never grow without limit by accident.
Q3What is the strongest honest claim after all fake-sink replay tests pass?
The tested software build produces the declared bounded outcomes for the captured fixtures and environment; live device, target timing, electrical behavior, and physical motion still require integration and hardware validation.
Chapter starter artifact
Replay bounded sensor outcomes
A clean-build gateway repository processes normal, malformed, stale, overload, saturation, sink-failure, replay, and cancellation cases with explicit ownership, bounded queues, structured events, timing tails, and no claim beyond the tested fake sink.
const mission = "night-shift-gateway";
const nowMs = 120;
const maxAgeMs = 50;
const commandLimit = 1;
const samples = [
{ stampMs: 110, value: 0.4, workMs: 4 },
{ stampMs: 100, value: 1.8, workMs: 7 },
{ stampMs: 20, value: 0.2, workMs: 12 },
];
let stale = 0;
let limited = 0;
const outcomes = samples.map((sample) => {
const ageMs = nowMs - sample.stampMs;
if (![sample.stampMs, sample.value, sample.workMs].every(Number.isFinite) ||
ageMs < 0 || sample.workMs < 0) throw new Error("invalid sample");
if (ageMs > maxAgeMs) {
stale += 1;
return { status: "STALE", command: 0 };
}
const command = Math.max(-commandLimit, Math.min(commandLimit, sample.value));
if (command !== sample.value) limited += 1;
return { status: "OK", command };
});
if (outcomes.length !== samples.length) throw new Error("lost replay input");
console.log("mission=" + mission);
console.log("processed=" + outcomes.length);
console.log("stale=" + stale);
console.log("limited=" + limited);
console.log("worstMs=" + samples.reduce((worst, sample) => Math.max(worst, sample.workMs), 0));
console.log("status=PASS");Download the file into your terminal's current folder, then run the command below. The expected output is exact.
- Run
node week-04-night-gateway.mjs
- Expected output
mission=night-shift-gateway processed=3 stale=1 limited=1 worstMs=12 status=PASS
- Planted failure to diagnose
Delete the source timestamp check. The 100 ms-old sample then produces a nonzero command, proving that successful parsing and bounded magnitude do not make stale evidence safe.