Chapter 24 · Train, optimize, evaluate, and freeze one bounded edge policy

Today in the field story

One problem, then the next

Model v1 reaches a Jetson-class deployment profile only if that target is actually available. You measure input age, preprocessing, inference, decoding, communication, p50 and tail latency, deadline misses, queues, memory, power mode, temperature, throttling evidence, and fallback under warm sustained load. Device and software identities accompany every result. A desktop timing number cannot substitute. If hardware is absent, the Forge preserves the profiling plan and marks target behavior unverified rather than inventing thermal or power claims.

Why now: Edge feasibility depends on the complete sustained path, not isolated inference speed.
Ignore today: Do not extrapolate across Jetson modules, power modes, drivers, or runtimes.
Unlocks next: A measured target envelope or a precise external-environment blocker.

Understand

Build the physical picture first

An edge computer is a backpack carried by the robot: weight, heat, energy, waiting lines, and stale messages matter as much as raw inference speed.

Profile the complete physical-information loop on the intended class of target. Observation age begins at sensor acquisition, then data may wait, copy, resize, normalize, enter the engine, decode, pass guards, travel through ROS 2, and reach command acceptance. Define stage latencies t_capture, t_queue, t_pre, t_infer, t_post, and t_transport; end-to-end age is their measured composition for one correlated request. GPU kernel time alone omits the queues and transformations most likely to make an otherwise accurate action stale.

Report distributions under representative load. p50 is a typical median sample, p95 is the value at or below which about 95 percent of measured samples fall, and deadline-miss rate counts samples beyond the application deadline. Warm up the actual shapes and paths, use realistic inputs, synchronize timing correctly, and separate inference-only from full-pipeline numbers. Test concurrent camera, recording, ROS 2, UI, or mapping work that will share CPU, memory bandwidth, and power; a quiet bench benchmark can be an optimistic special case.

Power and temperature create stateful performance. Pin the exact module, carrier, storage, JetPack or Jetson Linux release, TensorRT and CUDA versions, power mode, clocks policy, fan profile, enclosure, ambient temperature, and input load. Observe memory, temperatures, frequencies, power or rail signals available for that device, throttling events, and latency over a sustained interval. A five-second burst before the heat sink warms cannot establish a thirty-minute duty cycle, and maximum-performance mode is not automatically the permitted product operating mode.

Design failure before deployment. Give each inference a source timestamp, sequence, deadline, expiry, and cancellation lineage; bound queue depth and prefer dropping obsolete work over executing it late. On timeout, invalid output, lost executor, over-temperature policy event, memory pressure, or stale observation, the application may hold, cancel, request a known controller stop, fall back to a verified baseline, or require human handoff according to the task design. These are software responses, not E-stop, STO, protective stop, or a substitute for risk-derived safeguards.

Words you need

Name each idea precisely

End-to-end action age: Elapsed time from the physical observation's acquisition to the point where its derived command is accepted or rejected by the declared consumer.
Physical example:
A camera frame captured at 12:00:00.000 produces a guarded ROS 2 goal accepted at 12:00:00.084, so action age is 84 ms.
Tail latency: Latency near the slow end of a measured distribution, such as p95 or p99, which exposes rare delays hidden by an average.
Physical example:
Most grasp inferences finish near 48 ms, but one in twenty takes more than 112 ms when video encoding contends for memory.
Thermal throttling: Automatic performance reduction caused by a device's thermal management when monitored conditions reach implementation-specific limits.
Physical example:
After sustained inference warms the module, GPU frequency falls and p95 latency rises even though the engine and inputs have not changed.
Bounded queue: A waiting structure with a fixed capacity and explicit overflow policy so obsolete observations cannot accumulate without limit.
Physical example:
A latest-only camera queue replaces an unprocessed old frame, while the discarded sequence is counted and never converted into a late command.
Application fallback: A tested ordinary-software response to a named failure, with entry conditions, command authority, terminal state, observability, and escalation.
Physical example:
On inference timeout, the action server cancels the policy goal, commands the already validated hold behavior, and asks the operator to reset the task.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalLatency, memory, quantization, and acceptance deltas

A trained model is deployable only when it fits the edge device and preserves task quality.

FPS = 1/T: inferences per second from latency TUnit: frames/s
memory: runtime RAM or VRAM useUnit: MB or GB
Δmetric: optimized minus original metricUnit: metric unit

Inference latency is 50 ms = 0.05 s.
Maximum theoretical rate is 1/0.05 = 20 FPS.
Measure end-to-end rate under thermal load and compare the exact frozen success suite before and after ONNX/TensorRT or quantization.

Programmer analogy

Like profiling any application on its target device, optimize on the actual robot computer and protect behavior with regression tests.

What is the theoretical rate for 100 ms inference?

100 ms = 0.1 s; 1/0.1 = 10 FPS.

The policy budget is

T=\frac{1}{10}=100\ \mathrm{ms}.

Its p95 path is $18+62+6+9=95\ \mathrm{ms}$ , leaving

m_{p95}=100-95=5\ \mathrm{ms}.

Close a 100 ms action deadline with honest stage evidence

A warm nominal sample records capture and transfer 12 ms, queue 8 ms, preprocessing 9 ms, TensorRT inference 38 ms, postprocess plus guards 7 ms, and ROS 2 delivery plus acceptance 8 ms. The deadline is 100 ms, but a sustained 30-minute run reports p95 total age of 118 ms.

Sum the nominal stages: T_nom = 12 + 8 + 9 + 38 + 7 + 8 = 82 ms, leaving 100 - 82 = 18 ms nominal headroom.
Keep the measured p95 result separate from the nominal sum: 118 ms misses the deadline by 18 ms, proving the pipeline does not satisfy the declared tail gate.
Correlate slow samples with queue depth, temperature, GPU and CPU frequency, recording load, memory pressure, and input shape; do not blame inference until stage traces identify the delay.
Set every request's expiry to the action contract and make queue capacity explicit. When estimated or measured age exceeds 100 ms, reject the output before command dispatch and count deadline_expired.
Exercise the fallback: cancel the policy action, prevent queued goals from reviving, request the commissioned controller's ordinary hold or idle behavior, expose the reason, and require the designed reset or handoff.
Only after a development change, rerun the same warm-up, load, duration, input distribution, power mode, ambient condition, and acceptance calculation; never replace p95 with the attractive 82 ms sample.

Result

The current edge bundle is rejected by its latency gate even though one typical sample fits; the system expires late commands and demonstrates a bounded software fallback without making a safety-function claim.

What this proves

Deadline compliance is a distribution measured through the whole application under sustained target conditions, not the sum of one convenient trace or an engine benchmark alone.

Physical examples

Where this appears in real life

Fast engine behind an old camera queue

TensorRT inference takes 32 ms, but three 70 ms-old frames wait ahead of the current frame while the robot and block continue moving.

Look for:

Measure source age and queue time, cap the queue, expire old actions, correlate sequence IDs, and avoid presenting kernel latency as control-loop latency.

Cool demo becomes a hot shift

A Jetson-class device meets a 100 ms deadline for the first minute, then enclosure temperature rises and p95 latency reaches 138 ms during simultaneous recording.

Look for:

Use a sustained representative soak, record thermal and frequency evidence, test the approved power and cooling setup, and enter the declared degraded or fallback state on misses.

Hands-on exercise

Make the idea observable

Use the actual Jetson-class target when available or a clearly labelled constrained equivalent. Keep execution disconnected from actuators and replay representative observations through the complete application path.

Record exact hardware, software image, engine checksum, power and fan mode, enclosure or bench condition, ambient estimate, input shapes, concurrent services, queue policy, and measurement clock.
Add one correlation ID carrying acquisition time, queue entry, preprocessing start and end, inference start and end, decode and guard result, publish time, acceptance, expiry, and cancellation.
Warm the real shapes and code path, then profile inference-only and end-to-end latency separately across a declared sample count and at least one sustained representative-load interval.
Calculate p50, p95, p99 when the sample count supports it, maximum, deadline misses, dropped inputs, queue depth, memory high-water mark, and observed temperature, clock, throttling, or power signals.
Inject stale input, engine timeout or exception, full queue, executor loss, and a simulated thermal-policy event; verify that commands expire, cancellation propagates, fallback is bounded, and restart requires the declared conditions.
Compare the optimized candidate with the runnable baseline under identical load, then publish pass, revise, or reject against the experiment charter without claiming that an equivalent computer proves Jetson behavior.

Observe

Tail delay often appears outside the engine—in capture, copies, queuing, logging, memory contention, first-use shape work, or transport—and grows only after concurrent services and heat are present.

Done when

The target profile is reproducible, every sample has end-to-end lineage, sustained tail and thermal evidence meet the gate or visibly fail, and every injected fault reaches its tested application response.

Build today

Fine-tune one policy, export and profile it on Jetson-class hardware or an equivalent constrained target, deploy behind a safe ROS 2 action, and evaluate held-out scenes.

Evidence to save

DONE when the integrated “Edge deployment on Jetson-class hardware: latency, thermals, and fallback” path is observable, cancelable, and leaves the prior baseline reproducible.

Common mistakes

Catch the wrong mental model

Wrong

Reporting GPU compute time as the robot's action latency.

Better

Measure acquisition age through queue, preprocessing, inference, decoding, guards, transport, and command acceptance with one correlated monotonic timeline.

Wrong

Running ten cool samples in maximum-performance mode and calling thermals verified.

Better

Profile the approved module, software, power, fan, enclosure, ambient, concurrent load, and duty duration while recording temperature, clocks, throttling evidence, and tail latency.

Wrong

Calling a ROS 2 cancel callback or zero command a safety stop.

Better

Describe and test the ordinary application response precisely, while leaving E-stop, STO, protective stops, braking, and safety validation to the qualified risk-derived system.

Job connection

How this becomes employable evidence

Integrate and qualify an optimized policy on constrained edge compute by tracing sensor-to-command age, controlling queue and runtime state, profiling under realistic concurrent and thermal load, and connecting deadline, resource, and executor failures to observable cancellation and rollback behavior.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Deployment, Integration & Validation Engineer

Chapter 24 interview drill

Interview questions: Edge deployment on Jetson-class hardware: latency, thermals, and fallback

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A TensorRT benchmark reports 35 ms inference, but the robot occasionally reacts 200 ms late after warming up. Walk through the timestamps, queues, load, thermal evidence, percentiles, expiry policy, fallback test, and release decision you require.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What is the nominal end-to-end age in the worked example?

Model interview answer

12 + 8 + 9 + 38 + 7 + 8 = 82 ms, which is one nominal sample and does not override the measured 118 ms p95.

Q2Why should a full observation queue usually not keep growing?

Model interview answer

Old observations produce stale actions for a changed physical scene, so capacity, overflow, expiry, and discard telemetry must be explicit and bounded.

Q3What does a tested application fallback fail to prove?

Model interview answer

It does not prove an independently engineered safety function, safe stopping distance, E-stop, STO, protective-stop performance, or authorization for powered operation.