Chapter 24 · Train, optimize, evaluate, and freeze one bounded edge policy

Today in the field story

One problem, then the next

The chosen checkpoint crosses the export furnace with its processor and decoder attached. Structural ONNX validity, input and output shapes, representative golden values, action de-normalization, and physical tolerance checks remain distinct. If TensorRT or quantization is unavailable, you document the blocked gate rather than simulate a pass. Reduced precision is accepted only when source-versus-export drift stays inside predeclared downstream tolerances across the corpus, including boundary inputs and guard behavior.

Why now: An optimized artifact can be structurally valid yet semantically or physically different.
Ignore today: Do not claim TensorRT parity without the runtime or use one convenient input.
Unlocks next: A candidate engine or an explicit blocked optimization path with differential evidence.

Understand

Build the physical picture first

Export is translating a pilot's instructions into a smaller machine's language: every lever name, scale, order, and response must survive the translation.

Treat export as a chain of contracts, not a file conversion. The source policy includes preprocessing, tensor names, shapes, dtypes, camera order, normalization, history, padding, task encoding, recurrent or action-queue state, and postprocessing. ONNX represents a computation graph and operator set; TensorRT builds an executable engine for declared shapes, precision, software, and target compatibility. An ONNX checker can establish structural consistency, but it cannot prove that preprocessing still matches training or that output numbers mean the same physical commands.

Create a golden corpus G = {x_i} of representative development inputs: nominal scenes, allowed boundary shapes, dark and bright frames, near-limit states, padded histories, and deliberately invalid cases. For each valid x_i, keep source output y_i before decoding and compare ONNX or engine output ŷ_i. Define element error e_ij = ŷ_ij - y_ij, maximum absolute drift E_max = max |e_ij|, and mean absolute drift E_mean = mean |e_ij|. Tolerances must be per output and tied to downstream units, guards, and task sensitivity.

Reduced precision trades representation detail for resource use. FP16 has less range and precision than FP32; integer quantization maps values through scales, so rounding approximates values and clamping clips values outside the representable range. Post-training calibration inputs should resemble the intended training or development distribution and must not borrow sealed final cases. A good global error can still hide sign flips, saturated gripper values, rare NaN or Inf outputs, or a threshold crossing that changes a discrete command.

Advance in rungs: source framework, exported ONNX runtime, TensorRT FP32 or closest reference, then each lower-precision candidate. At every rung check graph, input contract, values, decoded actions, guard decisions, development rollout outcomes, memory, and latency. Reject or isolate unsupported operators instead of replacing them silently. The optimized engine remains behind the same freshness, bounds, rate, collision, authority, cancellation, and fallback path; speed does not grant motion authority or certify the controller.

Words you need

Name each idea precisely

Export contract: The complete agreement for model graph, operator set, tensor names, dimensions, dtypes, preprocessing, state, outputs, decoding, and supported runtime conditions.
Physical example:
The front camera must remain RGB, 1×3×480×640, normalized with training statistics, and paired with six ordered joint positions in radians.
Golden corpus: A versioned set of representative and boundary development inputs with preserved source outputs and expected validation behavior for differential testing.
Physical example:
Twenty recorded observations include centered blocks, edge poses, glare, padded histories, a near-limit wrist, and one rejected malformed image.
Numerical drift: The measured value difference between source and converted runtime outputs for the same exact inputs, precision, state, and decoding contract.
Physical example:
The TensorRT engine predicts a wrist delta 0.004 radians above the source output on one golden observation.
Quantization scale: A declared mapping between real-valued tensors and lower-precision representable values that determines rounding resolution and clipping range.
Physical example:
A scale maps gripper logits into INT8 levels; a poor range clips strong close commands and changes when the threshold is crossed.
Optimization profile: A TensorRT build-time declaration of permitted minimum, typical, and maximum runtime dimensions for inputs whose shapes vary.
Physical example:
An image-history input permits 2 to 8 frames, is optimized for 4, and must reject a 10-frame request rather than reinterpret memory.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalLatency, memory, quantization, and acceptance deltas

A trained model is deployable only when it fits the edge device and preserves task quality.

FPS = 1/T: inferences per second from latency TUnit: frames/s
memory: runtime RAM or VRAM useUnit: MB or GB
Δmetric: optimized minus original metricUnit: metric unit

Inference latency is 50 ms = 0.05 s.
Maximum theoretical rate is 1/0.05 = 20 FPS.
Measure end-to-end rate under thermal load and compare the exact frozen success suite before and after ONNX/TensorRT or quantization.

Programmer analogy

Like profiling any application on its target device, optimize on the actual robot computer and protect behavior with regression tests.

What is the theoretical rate for 100 ms inference?

100 ms = 0.1 s; 1/0.1 = 10 FPS.

Element-wise absolute drift is $[|0.50-0.49|,|-0.20-(-0.18)|]=[0.01,0.02]$ , so

d_{\max}=\max(0.01,0.02)=0.02.

Decide whether one lower-precision engine preserves action meaning

For one golden observation, the source policy outputs normalized action [0.120, -0.040, 0.501]. An FP16 TensorRT engine outputs [0.121, -0.043, 0.498]. Components decode to shoulder radians, wrist radians, and a gripper score thresholded at 0.5.

Compute signed errors as e = ŷ - y = [0.001, -0.003, -0.003]; keep signs because direction changes can matter even when magnitudes match.
Compute E_max = 0.003 and E_mean = (0.001 + 0.003 + 0.003) / 3 ≈ 0.00233; label the first two components in normalized units until decoder scales are applied.
Apply component tolerances: shoulder 0.002, wrist 0.002, and gripper decision equality. The wrist exceeds its numeric tolerance because 0.003 > 0.002.
Decode both gripper scores: source 0.501 ≥ 0.5 means close, while engine 0.498 < 0.5 means open; semantic parity fails despite the small global error.
Repeat across the entire golden corpus and group errors by output, scene factor, value magnitude, invalid input, and guard decision; inspect NaN, Inf, saturation, and sign changes explicitly.
Reject this FP16 engine for release, retain the source or higher-precision rung, and investigate mixed precision, threshold robustness, calibration coverage, or architecture support only on development data.

Result

The engine fails both a wrist tolerance and the gripper decision contract, so a tempting speed improvement cannot advance until a new candidate passes the unchanged parity suite.

What this proves

Numerical acceptability is defined by downstream physical meaning; one averaged drift number cannot overrule a changed discrete action or a safety-relevant component.

Physical examples

Where this appears in real life

Correct graph, swapped cameras

The ONNX checker passes and tensor shapes match, but deployment supplies the wrist camera under the front-camera name and the front image under the wrist key.

Look for:

Structural validity cannot detect semantic exchange between equal-shaped tensors. Golden observations, named binding checks, image fingerprints, and action parity must exercise camera identity.

Tiny float change crosses a gripper threshold

The source produces gripper score 0.501; a quantized engine produces 0.498. Absolute drift is only 0.003, yet a 0.5 threshold changes close into open.

Look for:

Set tolerance from physical decision behavior, compare postprocessing and guard results, and consider keeping the sensitive layer or decision in higher precision.

Hands-on exercise

Make the idea observable

Use the selected checkpoint, its exact processors and decoder, ONNX tooling, and TensorRT when available. If the target stack is unavailable, create the contracts and run source-versus-ONNX tests without pretending TensorRT was verified.

Write a manifest for every input and output with name, order, dtype, shape or range, unit before and after decoding, preprocessing, state ownership, freshness, and invalid-input response.
Build a versioned golden corpus from development data only, preserving raw inputs, source preprocessed tensors, source outputs, decoded actions, and expected guard decisions.
Export with a pinned operator-set and shape policy, run the ONNX checker and shape checks, then load the artifact in a fresh runtime and reject missing or reordered bindings.
Compare source, ONNX, TensorRT reference precision, and each quantized candidate on identical bytes; calculate per-component max, mean, percentiles, sign flips, threshold changes, NaN, Inf, and saturation.
Run the unchanged development rollout suite for every numerically eligible rung, recording task outcomes, guard rejections, memory, warm and steady latency, and any unsupported operator or fallback.
Package the exporter, build command, calibration lineage, compatibility record, logs, model and engine checksums, and explicit pass or reject table without overwriting the source checkpoint.

Observe

The first failure may be semantic rather than numeric: color order, camera binding, dynamic history length, padding state, or decoder statistics can be wrong while the graph remains legal.

Done when

Every runtime has exact bindings and provenance, structural checks pass, value and semantic tolerances pass on all golden cases, development behavior remains eligible, and no sealed final input entered export or calibration.

Build today

Fine-tune one policy, export and profile it on Jetson-class hardware or an equivalent constrained target, deploy behind a safe ROS 2 action, and evaluate held-out scenes.

Evidence to save

DONE when a deterministic “Export to ONNX/TensorRT, quantize, and verify numerical drift” failure test reports expected versus actual behavior and passes after the documented fix.

Common mistakes

Catch the wrong mental model

Wrong

Treating onnx.checker.check_model() as proof that the exported policy behaves like the source.

Better

Use the checker for graph legality, then separately verify bindings, preprocessing, state, numerical outputs, decoding, guard decisions, rollout outcomes, and timing.

Wrong

Accepting quantization because average output drift is small.

Better

Measure every action component, worst and tail error, sign and threshold changes, saturation, invalid values, and closed-loop development outcomes against physically justified tolerances.

Wrong

Using held-out final scenes as representative INT8 calibration examples.

Better

Build calibration coverage from training or declared development data; final cases must remain unavailable to every optimization choice, including precision calibration.

Job connection

How this becomes employable evidence

Own a framework-to-edge conversion gate that inventories preprocessing and tensor semantics, diagnoses unsupported operators, constructs representative calibration and golden corpora, performs source-versus-engine differential tests, and rejects optimizations that change action or guard behavior.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Deployment, Integration & Validation Engineer

Chapter 24 interview drill

Interview questions: Export to ONNX/TensorRT, quantize, and verify numerical drift

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

An ONNX model passes the checker and a TensorRT INT8 engine is twice as fast. Explain why neither fact proves parity, which tensors and cases you compare, how you set physical tolerances, where calibration data comes from, and what blocks release.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What does an ONNX structural check fail to establish?

Model interview answer

It does not establish correct preprocessing, tensor identity, numerical parity, action semantics, runtime timing, task success, guard behavior, or physical safety.

Q2Why does `E_max = 0.003` not automatically pass the worked example?

Model interview answer

The wrist tolerance is 0.002, and the gripper's 0.003 shift crosses its 0.5 decision threshold, so physical semantics change.

Q3Where may post-training calibration examples come from?

Model interview answer

From representative training or explicitly available development data with recorded lineage, never from a suite still claimed to be untouched final evidence.