Chapter 04 · Production C++, Linux, and deterministic timing
Today in the field story
One problem, then the next
The gateway reads successfully, yet the controller misses its deadline during a logging burst. Measure the Night-Shift Sensor Gateway with a monotonic clock, absolute schedule, bounded queue, and explicit cancellation. Record period, execution, source age, tail, maximum, and miss count under load; then diagram the lock path that could create priority inversion before assigning any elevated scheduling priority.
- Why now
Robot usefulness depends on bounded arrival, processing, and shutdown—not average speed.
- Ignore today
Ignore hard-real-time claims and machine-wide scheduler changes.
- Unlocks next
Evidence for callback budgets, watchdog thresholds, and real-time risk reviews.
Understand
Build the physical picture first
A control loop is a bus with a timetable: finishing eventually is not enough—the useful result must arrive before its physical deadline, even while other work competes for the road.
Frequency says how many loop cycles are requested each second. Period is the time between ideal cycle starts and equals one divided by frequency. A 100 Hz loop has a 0.01 s, or 10 ms, period. Execution time is how long that cycle's work consumes; end-to-end latency is the age from a physical event or sample timestamp to the resulting action or observation. The deadline is the latest useful completion time. These values are related but not identical: work taking 3 ms can still act on a sample that waited 40 ms in a queue.
Jitter is variation around expected timing. Start-time jitter compares actual starts with the planned schedule; execution-time jitter compares how long repeated work takes. Report a distribution or at least minimum, median, high percentile, maximum, and miss count under a declared load. An average of 4 ms does not protect a 10 ms deadline if occasional cycles take 18 ms. Use a monotonic clock for intervals because wall-clock time may jump when synchronized or manually changed.
Threads let pieces of one process make progress concurrently and share memory, but they do not promise simultaneous execution or deterministic order. A mutex protects an invariant by allowing one thread at a time into a critical section. Keep that section bounded and do not perform unknown blocking I/O while holding the lock. A queue between sensor and controller must also be bounded: if input arrives faster than processing, decide whether to reject, drop oldest for freshness, drop newest for continuity, or enter a safe state. Letting memory grow is not a timing policy.
Real-time scheduling means bounded response is the design goal; it does not mean simply fast. On Linux, ordinary tasks commonly use fair scheduling, while real-time policies can let higher-priority runnable work preempt lower-priority work. Priority inversion occurs when a high-priority thread needs a lock held by a lower-priority thread and is indirectly delayed by medium-priority work. Priority inheritance can temporarily raise the lock owner's priority, but it cannot repair unbounded work, page faults, bad drivers, blocking calls, overload, or incorrect deadlines. First bound the design, then measure on the target under representative load.
Words you need
Name each idea precisely
- Period
The intended time between cycle starts; for frequency f, period T equals 1/f.
Physical example:A 50 Hz safety monitor starts a new check every 20 ms.
- Deadline
The latest time by which work must complete to remain useful for its physical purpose.
Physical example:A wheel command must be refreshed before the watchdog's 100 ms expiry.
- Jitter
Variation of observed start, completion, or latency values around the expected timing.
Physical example:A nominal 10 ms loop begins after gaps of 9.7, 10.1, and 12.4 ms.
- Mutex
A synchronization object that permits one owner at a time to protect a shared invariant.
Physical example:The state estimator and monitor do not modify one shared status record at the same time.
- Priority inversion
A higher-priority task is blocked by a resource held by lower-priority work, potentially extended by medium-priority work.
Physical example:A control thread waits for a diagnostics thread's lock while an unrelated worker repeatedly preempts the lock holder.
Visual model
See the relationship
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalRates, deadlines, jitter, and memory budgets
Hardware-facing software must finish work predictably before its deadline.
- f
- loop frequencyUnit: hertz (Hz)
- T = 1/f
- time available for one loopUnit: seconds (s)
- jitter
- variation around the expected timingUnit: milliseconds (ms)
For a 100 Hz control loop, T = 1/100 s.
Convert: 0.01 s = 10 ms per cycle.
If work sometimes takes 13 ms, it misses the 10 ms deadline by 3 ms; measure the distribution, not only the average.
This is a strict rendering or audio-processing budget, except a missed robot deadline can destabilize an actuator.
How much time does a 50 Hz loop have per cycle?
1/50 s = 0.02 s = 20 ms.
The loop period is
Relative to ideal starts at , , and , the jitter values are , , and ; execution time misses the deadline.
Compute timing evidence for a 100 Hz loop
Ideal starts are 0, 10, 20, 30, and 40 ms. Actual starts are 0.0, 10.4, 19.7, 31.2, and 40.1 ms. Execution times are 4, 5, 12, 6, and 4 ms.
Convert frequency to period: T=1/100 s=0.01 s=10 ms.
Subtract each ideal start from its actual start to get start jitter: 0.0, +0.4, -0.3, +1.2, and +0.1 ms.
Compare each execution time with the 10 ms cycle budget: the 12 ms cycle exceeds that budget by 2 ms.
For that cycle, compute completion at 19.7+12=31.7 ms and compare it with the absolute 30 ms deadline, so it finishes 1.7 ms late.
Count deadline misses: 1 miss among 5 observed cycles, or 20% in this tiny sample.
Report the maximum observed start jitter magnitude as 1.2 ms and maximum execution time as 12 ms.
State the missing evidence: five unloaded samples cannot establish a production timing bound; run long tests under declared CPU, I/O, thermal, and network load.
The mean-looking loop is not acceptable for a strict 10 ms deadline because at least one measured execution exceeds the budget.
A rate request is not timing proof; timestamps, tail values, miss counts, load, and target configuration are the proof.
Physical examples
Where this appears in real life
Conveyor camera deadline
A camera identifies an item correctly, but the result reaches the diverter after the item has already passed.
Accuracy is irrelevant for that item once capture, queue, inference, and command latency exceed the physical travel deadline.
Three workers and one calibration notebook
An urgent safety worker needs a notebook held by a low-priority recorder, while a medium-priority reporter keeps interrupting the recorder.
The urgent worker is indirectly delayed; priority inheritance helps the recorder finish and release the shared notebook.
Hands-on exercise
Make the idea observable
Write a harmless C++ timing program with a periodic worker and a background load thread; do not request real-time privileges or change machine-wide scheduling settings.
Use a monotonic clock and schedule ideal starts at a 20 ms period.
At each cycle, record ideal start, actual start, execution completion, and a cycle sequence number.
Add bounded synthetic work and make every twentieth cycle deliberately longer than the budget.
Run a background thread that performs bounded CPU work without sharing the timing log during the measured section.
Compute start jitter, execution time, maximum, a high percentile if enough samples exist, and deadline-miss count.
Add a bounded stop flag or stop token, request cancellation, join both threads, and measure shutdown time.
Repeat with and without background load and save both configurations beside the results.
The requested sleep period, actual start schedule, execution budget, and cancellation time differ; load and deliberate outliers appear in tail metrics.
At least 500 cycles produce machine-readable timing records, the planted overruns are counted, shutdown is bounded, and the report states clock, load, period, and hardware used.
Build today
Create a Linux C++ sensor→filter→controller pipeline with device permissions, bounded timing, tests, CI, structured logs, and deterministic replay.
Evidence to save
DONE when a comparison table for “Threads, real-time scheduling, jitter, and priority inversion” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Calling a loop real-time because its average execution is below the period.
Check end-to-end deadlines, tail and maximum timing, miss count, queue age, and representative load on the target.
Using repeated relative sleeps as the timing schedule.
Schedule against absolute monotonic deadlines so work time and wake-up error do not silently accumulate as drift.
Giving the control thread the highest priority before bounding its work.
Remove unbounded blocking, constrain queues and locks, define cancellation, then measure; an unbounded high-priority loop can starve the whole system.
Job connection
How this becomes employable evidence
Measure a driver or control callback under representative load, bound its queue and critical sections, and expose stale data and deadline misses to validation and operator tooling.
Relevant target roles
- Robotics Software Engineer — ROS 2 / AMR
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robot HMI / Control & Monitoring Engineer
Chapter 04 interview drill
Interview questions: Threads, real-time scheduling, jitter, and priority inversion
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A 100 Hz loop averages 3 ms but still oscillates under load. Explain which timestamps and tail metrics you need, how queues and locks can add latency, and why increasing thread priority is not the first complete fix.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1How much time is available per cycle at 50 Hz?
T=1/f=1/50 s, so the ideal interval between cycle starts is 0.02 s, or 20 ms.
Q2Why can 3 ms execution still produce a 40 ms-old command?
The sample may have waited in a queue or earlier pipeline stages; execution time measures only one section, while end-to-end latency includes waiting and upstream work.
Q3What does priority inheritance do during priority inversion?
It temporarily gives the lock holder the blocked higher-priority task's priority so the holder can run and release the lock sooner; it does not make the whole design deterministic.