Chapter 24 · Train, optimize, evaluate, and freeze one bounded edge policy

Today in the field story

One problem, then the next

The Forge freezes Model v1 as a system bundle: checkpoint, processor, statistics, schema, adapter, exported graph or engine recipe, runtime profile, command guard, ROS interface, target identity, experiment charter, parity corpus, held-out report, monitoring fields, fallback, rollback, approvals, and limitations. Checksums replace labels like latest. A rejected candidate may still be frozen as evidence but cannot be promoted. The bundle and exact verdict move to Week 25, where service-level deployment and acceptance must preserve every identity.

Why now: Operations can reproduce and roll back only a complete immutable bundle.
Ignore today: Do not publish an isolated weights file as a deployable robot system.
Unlocks next: A release candidate with enough identity for staged operational evidence.

Understand

Build the physical picture first

A deployable model is a labelled spare engine in a service crate: the exact parts, fit, limits, test sheet, installation recipe, and rollback route travel together.

Freeze a system bundle, not a lonely weights file. The bundle includes source checkpoint, policy configuration, processors, tokenizers when present, normalization and de-normalization statistics, observation and action schemas, robot adapter, ONNX graph, TensorRT engine or reproducible build recipe, calibration lineage, runtime dependencies, command guard, ROS 2 interface, target profile, and checksums. Name the exact selected artifact at every step; “ACT v1” or “latest engine” cannot tell an operator which bytes produced a command.

Write an operating envelope from measured evidence. State supported robot and calibration identities, task, objects and scenes, camera configuration, input shapes, rates, action mode, units, frames, limits, compute module, software image, power and cooling condition, concurrency, latency deadline, queue policy, startup checks, fallback, and known unsupported conditions. The held-out report establishes only the tested slices and counts. It does not license novel tools, payloads, firmware, lighting, enclosures, or autonomous operation beyond the approved risk controls.

Make release gates machine-checkable where practical. On installation, verify hashes, signatures or trusted provenance, compatibility, schemas, processors, golden-corpus parity, engine load, representative latency smoke, guard behavior, cancellation, fallback, monitoring, and rollback. During operation, observe version, input freshness, queue age, inference and end-to-end latency, invalid values, guard reasons, action expiry, executor state, resource pressure, and application fallbacks without logging sensitive raw data unnecessarily. A heartbeat or successful load is availability evidence, not task acceptance.

Practice restoration before calling v1 deployable. Keep the previous accepted baseline immutable, document how to select it, and rehearse rollback in the disconnected or approved staging path after injected load, parity, latency, and startup failures. Publish evaluation, drift, latency and thermal reports, deviations, approvals, limitations, and a first post-release experiment. If any required artifact, target test, owner, or safety boundary remains missing, label the bundle candidate or blocked; a deadline does not convert incomplete evidence into release.

Words you need

Name each idea precisely

Deployment bundle: The checksummed collection of model, processing, schema, adapter, runtime, guard, target, evidence, monitoring, fallback, and rollback artifacts required to reproduce behavior.
Physical example:
A service package pairs one engine plan with camera ordering, joint statistics, robot calibration ID, Jetson image, latency report, and previous-version restore command.
Operating envelope: The bounded physical, sensing, compute, software, timing, and task conditions for which the collected evidence supports use.
Physical example:
Model v1 supports one foam-block task, fixed cameras, declared lighting range, 10 Hz actions, one robot calibration, and the tested power and cooling mode.
Artifact provenance: Traceable origin and transformation history linking each released byte to source code, data, configuration, builder, environment, reviewer, and evidence.
Physical example:
The engine checksum points to an ONNX checksum, exporter revision, TensorRT version, build profile, calibration manifest, and signed release record.
Rollback: A rehearsed transition from a failing candidate to a known prior bundle or disabled state, including trigger, authority, compatibility, verification, and terminal result.
Physical example:
A latency smoke-test failure prevents activation, restores baseline b7, confirms its golden outputs, and records why model v1 stayed inactive.
Release gate: A mandatory, evidence-linked condition that an identified bundle must pass before a named environment and authority may activate it.
Physical example:
The exact engine cannot enter staging unless hash, schema, parity, p95 latency, cancellation, fallback, and evaluation checks all pass.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalLatency, memory, quantization, and acceptance deltas

A trained model is deployable only when it fits the edge device and preserves task quality.

FPS = 1/T: inferences per second from latency TUnit: frames/s
memory: runtime RAM or VRAM useUnit: MB or GB
Δmetric: optimized minus original metricUnit: metric unit

Inference latency is 50 ms = 0.05 s.
Maximum theoretical rate is 1/0.05 = 20 FPS.
Measure end-to-end rate under thermal load and compare the exact frozen success suite before and after ONNX/TensorRT or quantization.

Programmer analogy

Like profiling any application on its target device, optimize on the actual robot computer and protect behavior with regression tests.

What is the theoretical rate for 100 ms inference?

100 ms = 0.1 s; 1/0.1 = 10 FPS.

The release metrics remain separate:

S=\frac{46}{50}\times100\%=92\%,\qquad t_{p95}=88\ \mathrm{ms}<100\ \mathrm{ms},\qquad N_{\text{safety violations}}=0.

Freeze model v1 as a reviewable release candidate

Candidate checkpoint step_6000 passed source-to-engine parity and its development gates. Final evaluation reports 21 unassisted successes in 24 held-out trials, zero prohibited outcomes, and p95 end-to-end age of 91 ms against a 100 ms gate. A prior baseline remains available.

Assign immutable identities and checksums to checkpoint, configuration, processors, statistics, schema, adapter, ONNX, engine, builder recipe, guard, target image, dataset manifests, scorer, and every report.
Write the evidence ledger: parity corpus and tolerances pass, final success is 21/24 = 87.5%, prohibited count is zero, and p95 age 91 ms ≤ 100 ms; retain slice counts, interventions, failures, and thermal conditions.
Encode the operating envelope and unsupported conditions, including exact robot and calibration, cameras, task objects, action units and frames, target module, software, power and cooling, concurrent load, queue, deadline, fallback, and supervision.
Create startup gates for provenance, compatibility, bindings, golden outputs, finite decoded actions, representative latency, guard and cancel behavior, plus monitoring rules for age, misses, invalid output, resource pressure, and fallback.
Inject a bad checksum and a latency-smoke failure in the disconnected release path; require activation denial, execute rollback to the prior baseline, verify its identity and golden cases, and capture the restore duration and result.
Obtain required technical, validation, operations, and safety-boundary approvals for the stated environment, publish limitations and next experiment, and label the bundle v1 only if every mandatory owner and artifact closes.

Result

Model v1 becomes a reproducible, evidence-scoped deployment bundle with a tested refusal and rollback path, rather than an optimized file whose surrounding assumptions live in people's memories.

What this proves

Freezing means making the exact behavior reconstructable and bounded; it never means the model is universally finished or allowed to bypass physical safeguards.

Physical examples

Where this appears in real life

Engine file arrives without its measuring tape

A technician receives policy.plan, but not the camera order, normalization statistics, input profile, TensorRT version, robot calibration, or output decoder.

Look for:

The file is not deployable because equal-shaped inputs and outputs can carry wrong physical meaning. Activation stays blocked until the complete versioned bundle is restored.

Rollback exists only on a slide

A new edge image fails its latency smoke test. The documented baseline command references a removed container tag and cannot load the previous engine.

Look for:

A rollback claim requires an executable restore drill, retained artifacts, compatible configuration and data, startup verification, owner authority, and recorded terminal state.

Hands-on exercise

Make the idea observable

Use a clean disconnected workspace or approved staging image, the selected artifacts, prior baseline, and the week's evidence. Do not connect the bundle to unapproved hardware.

Generate a release manifest listing every artifact, checksum, producer revision, dependency, transformation, compatibility condition, evidence link, owner, and retention location.
Install the bundle into a clean process or image from the manifest alone, then validate schemas, processors, input profiles, golden outputs, decoded units, guards, cancellation, and a representative latency smoke.
Write the operating envelope and explicit non-claims in language an integrator can test, including physical task limits, compute and thermal conditions, supervision, fallback, and safety boundary.
Configure structured monitoring for artifact identity, observation age, queue depth, stage and tail latency, deadline misses, invalid values, guard decisions, action expiry, executor health, and fallback result.
Inject incompatible schema, corrupted checksum, stale observation, engine exception, missed deadline, and unavailable dependency; verify activation denial or bounded fallback with no hidden substitute.
Restore the prior baseline from retained artifacts, rerun its golden and startup checks, measure restore time, then publish the signed evaluation, drift, performance, limitation, rollback, and ship-or-block record.

Observe

The clean install exposes assumptions that a training machine hides: local caches, unpinned processors, missing statistics, target-built engines, renamed topics, unavailable plugins, or a rollback image that no longer starts.

Done when

A reviewer can recreate the bundle from immutable provenance, all gates and faults produce the declared result, rollback succeeds, the envelope matches evidence, and no missing approval is concealed by the v1 label.

Build today

Fine-tune one policy, export and profile it on Jetson-class hardware or an equivalent constrained target, deploy behind a safe ROS 2 action, and evaluate held-out scenes.

Evidence to save

DONE when the weekly ship note explains how “Freeze deployable model v1 with evaluation and latency report” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Publishing weights and an engine under the tag latest and calling that a release.

Better

Freeze immutable identities for the full processing, schema, adapter, runtime, guard, target, evidence, monitoring, fallback, and rollback bundle.

Wrong

Describing held-out success as support for any object, lighting, payload, or robot.

Better

Write the operating envelope from exact tested cases, hardware, software, timing, and conditions, and list unsupported or unknown conditions explicitly.

Wrong

Documenting rollback without running it from retained artifacts.

Better

Inject an activation failure, restore the prior bundle in a clean path, verify identity and golden behavior, measure the transition, and preserve the terminal evidence.

Job connection

How this becomes employable evidence

Own the final robot-policy release pack across model, processor, export, edge runtime, ROS 2 integration, observability, evidence, operating limits, and rollback; block activation on drift, compatibility, deadline, fallback, or approval failures and make field diagnosis version-exact.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 24 interview drill

Interview questions: Freeze deployable model v1 with evaluation and latency report

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

What belongs in a deployable robot-model v1 beyond weights? Describe provenance, processor and action contracts, target compatibility, operating envelope, startup and runtime gates, evaluation scope, fault injection, monitoring, rollback rehearsal, and safety non-claims.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What is the held-out success rate in the worked example?

Model interview answer

21/24 = 0.875, or 87.5 percent, within the exact tested cases and not beyond the stated operating envelope.

Q2Why is a serialized engine alone not a deployment bundle?

Model interview answer

It omits input and output semantics, processors, statistics, target compatibility, guards, runtime, evidence, monitoring, fallback, rollback, provenance, and physical operating limits.

Q3What must happen when a mandatory approval is missing?

Model interview answer

The artifact remains a candidate or blocked for that environment; successful tests and schedule pressure do not create the missing authority.

Chapter starter artifact

Block a faster candidate that produces a prohibited outcome

Freeze one checksummed Model v1 bundle whose experiment charter, baseline comparison, checkpoint rule, export parity, sustained edge profile, untouched held-out verdict, fallback, rollback, and operating limits can be reproduced.

week-24-model-v1-decision.mjsJavaScriptDownload starter

const metricFields = ["success", "p95Ms", "prohibited"];
const gates = { minimumSuccess: 0.8, maximumP95Ms: 100, maximumProhibited: 0 };
const candidate = { success: 0.85, p95Ms: 92, prohibited: 0 };
const plantedFailure = { ...candidate, p95Ms: 70, prohibited: 1 };
const missingMetric = { success: 0.85, prohibited: 0 };
const negativeMetric = { ...candidate, p95Ms: -1 };
const nonfiniteMetric = { ...candidate, success: Number.NaN };

function decide(result) {
  const missing = metricFields.find((field) => result?.[field] === undefined);
  if (missing) return "REJECT missing_" + missing.replace(/[A-Z]/g, (c) => "_" + c.toLowerCase());
  const invalid = metricFields.find(
    (field) => !Number.isFinite(result[field]) || result[field] < 0,
  );
  if (invalid) return "REJECT invalid_" + invalid.replace(/[A-Z]/g, (c) => "_" + c.toLowerCase());
  if (result.success > 1 || !Number.isInteger(result.prohibited)) return "REJECT metric_domain";
  if (result.prohibited > gates.maximumProhibited) {
    return "REJECT prohibited_outcome";
  }
  if (result.success < gates.minimumSuccess) return "REJECT success";
  if (result.p95Ms > gates.maximumP95Ms) return "REJECT latency";
  return "ACCEPT";
}

console.log("candidate gates: " + decide(candidate));
console.log("planted unsafe candidate: " + decide(plantedFailure));
console.log("missing metric: " + decide(missingMetric));
console.log("negative metric: " + decide(negativeMetric));
console.log("nonfinite metric: " + decide(nonfiniteMetric));
console.log("model v1: REJECT");

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-24-model-v1-decision.mjs
Expected output: candidate gates: ACCEPT planted unsafe candidate: REJECT prohibited_outcome missing metric: REJECT missing_p95_ms negative metric: REJECT invalid_p95_ms nonfinite metric: REJECT invalid_success model v1: REJECT
Planted failure to diagnose: A prohibited outcome, missing metric, negative latency, or non-finite success value blocks release instead of passing through JavaScript comparison edge cases.