Chapter 19 · Adapt a current VLA through explicit data, action, and release contracts

Today in the field story

One problem, then the next

The Policy Passport Review closes with an accounting session. You join artifact identities, retained data, exclusions, measured compute or wall time, observed latency, licensing or access constraints, regressions, guard behavior, and every unresolved risk. Unmeasured energy or cloud cost stays unknown rather than becoming a precise estimate. The board issues deploy, revise, or reject only for the tested simulator scope. Its final attachment is the guarded proposal contract that Courier-17 will receive in Week 20, never a claim that the model owns execution authority.

Why now: A release decision must combine capability, cost, reproducibility, and risk evidence.
Ignore today: Ignore production authorization and unsupported cost or energy precision.
Unlocks next: A policy proposal with enough provenance for bounded agentic execution.

Understand

Build the physical picture first

An adaptation report is the policy's release passport: it binds what changed to what it cost, where it worked, where it failed, and which exact gate controls entry.

Start the report with identity and decision scope. Name the base checkpoint and revision, adapter or fine-tuned checkpoint, framework and runtime, processor and statistics artifacts, robot and embodiment profile, dataset and split versions, source licenses, code revision, configuration, and evaluation manifest. State whether the result concerns offline inference, simulation, supervised restricted-energy trials, or a production candidate. A report that says “fine-tuned π0.5” or “tested GR00T” without these identities cannot be reproduced or safely paired.

Account for data and compute with measured values and explicit unknowns. Report raw and retained episodes, duration, tasks, embodiments, outcomes, held-out groups, exclusions, and sampling exposure. Record hardware, precision, batch and accumulation, trainable and total parameters, wall time, peak memory, checkpoint storage, and electrical energy only if measured or bounded from a declared power assumption. Cloud invoice, local electricity, engineering time, data collection, annotation, and deployment hardware are different cost categories and should not be collapsed into one invented number.

Connect model metrics to operational evidence. Place baseline and candidate side by side for offline loss or action error, familiar and held-out rollout success, intervention and abstention, prohibited behavior, guard rejection, p50 and p95 end-to-end latency, deadline misses, memory, startup time, and soak stability. Link every aggregate to raw trial IDs and failure artifacts. Include regressions and negative results prominently; average improvement cannot compensate for a missed mandatory safety, action-validity, or recovery threshold.

End with a reproducible gate and an owner. “Deploy” names the exact artifact, robot profile, environment, limits, monitoring, rollback, expiry, and approving roles. “Revise” lists the failed thresholds, corrective experiment, and evidence still needed. “Reject” preserves why the candidate is unsuitable rather than deleting the run. Unknowns stay unknown, marketing claims remain outside local evidence, and every future change to model, processor, data, adapter, controller, or site triggers the defined amount of revalidation.

Words you need

Name each idea precisely

Experiment lineage: The complete version chain linking source data, transforms, configuration, code, base, adapter, processors, runtime, evaluation, and report.
Physical example:
A failed trial points back to dataset v12, transform r3, OpenVLA base hash B, adapter A7, processor P4, and simulator case N-08.
Policy tail latency: A high-percentile end-to-end delay such as p95 that exposes slow observations or commands hidden by an average.
Physical example:
Median inference fits the control period, but p95 observation-to-command time misses it in one of twenty cycles.
Release regression: A previously accepted capability or constraint that performs worse under the candidate artifact or changed system.
Physical example:
The adapter improves novel cups but increases wrong-gripper actions on the old tray task.
Artifact acceptance gate: A threshold and decision rule fixed before final results that blocks promotion when any mandatory condition fails.
Physical example:
Release requires zero prohibited commands, p95 below 100 ms, and at least 18/20 successes in every mandatory task stratum.
Reproducibility manifest: A machine-readable inventory of artifacts, versions, environment, commands, seeds, and checksums needed to reconstruct a run and its evaluation.
Physical example:
A reviewer reruns case N-08 from the recorded container, policy bundle, dataset snapshot, and simulator reset file.
Deployment decision: An owned deploy, revise, or reject outcome whose scope, evidence, restrictions, monitoring, and rollback are explicit.
Physical example:
The adapter is approved only for the simulated cup cell, while hardware deployment remains blocked by latency and supervised-trial evidence.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalFine-tuning size, normalization, and evidence

Model adaptation must fit compute limits and improve frozen physical trials.

P_train: parameters updated during tuningUnit: parameters
GB: memory footprintUnit: gigabytes
Δsuccess: new minus baseline success rateUnit: percentage points

A baseline succeeds 12/20 = 60%; adaptation succeeds 16/20 = 80%.
Improvement is 80%−60% = 20 percentage points, not 20 percent.
Report parameter count, memory, latency, and the same frozen scenarios before claiming improvement.

Programmer analogy

Treat a policy release like a mobile release: same acceptance suite, device budget, rollback path, and versioned artifact.

A metric rises from 50% to 65%. What is the percentage-point gain?

15 percentage points.

The upper energy estimate is

E=Pt=(0.300\ \mathrm{kW})(4\ \mathrm{h})=1.2\ \mathrm{kWh}.

The tail-latency change is

\Delta t_{p95}=230-180=50\ \mathrm{ms}.

Turn a mixed adaptation result into a release decision

A candidate improves overall fixed-suite success from 28/40 to 33/40. Novel-object success is 14/20 against a gate of 16/20, one prohibited action occurs against a zero-tolerance gate, and p95 latency rises from 90 ms to 125 ms against a 100 ms deadline.

Verify both candidates used the same forty case IDs, resets, processor family, robot profile, terminal rules, and complete denominators; link every result to raw artifacts.
Calculate overall rates: baseline 28 / 40 = 70% and candidate 33 / 40 = 82.5%, an increase of 12.5 percentage points.
Evaluate mandatory strata separately: candidate novel-object success 14 / 20 = 70% misses the required 16 / 20 = 80% threshold.
Apply the prohibited-action gate: one observed event exceeds the allowed zero, so no average score can authorize promotion.
Calculate the tail miss: 125 ms - 100 ms = 25 ms, then record associated stale-output or fallback evidence and compare the 35 ms regression from the 90 ms baseline.
Issue REVISE, retain the candidate artifact, name the novel-object, command-guard, and latency investigations, and prohibit deployment until a newly versioned candidate passes the original untouched gate or an independently justified new release plan.

Result

The report preserves the useful 12.5-point aggregate improvement while correctly blocking release on three independently failed mandatory conditions.

What this proves

A decision rule exists to survive attractive results: improvement is informative, but every non-negotiable action, stratum, and deadline gate still applies.

Physical examples

Where this appears in real life

Better average, failed mandatory slice

The adapter raises overall success from 70% to 82% but produces one prohibited gripper action and drops novel-object success below the frozen minimum.

Look for:

The report rejects promotion under unchanged mandatory gates even though the headline average improved.

Fast demo, slow tail

A policy appears responsive during a short video, yet its measured p50 observation-to-command latency is 70 ms and p95 is 145 ms against a 100 ms deadline.

Look for:

The 45 ms p95 miss, queue behavior, fallback count, and sustained-load trace are decision evidence; the edited demo is not.

Hands-on exercise

Make the idea observable

Use saved outputs from this week's dry runs or a clearly labelled synthetic result set. Do not manufacture missing measurements and do not present paper or simulator results as hardware acceptance.

Create an identity page containing artifact hashes, versions, access dates, code, configuration, processors, statistics, robot profile, data lineage, licenses, and evaluation manifest.
Reconcile raw, retained, train, validation, and final episode counts; report duration, tasks, embodiments, sampling exposure, exclusions, and every held-out factor.
Record measured training and runtime hardware, precision, trainable parameters, time, peak memory, storage, latency, and energy or financial fields, marking unavailable values unknown.
Build paired baseline-candidate tables for offline, familiar, held-out, intervention, abstention, prohibited action, guard, latency, memory, and soak results with links to failures.
Apply predeclared gates without changing thresholds, write the deploy, revise, or reject outcome, and assign each unresolved item an owner and required evidence.
Have another engineer reconstruct one metric and one failure from the manifest, then rehearse rollback or adapter disablement at the same non-hardware evidence level.

Observe

A complete ledger makes unsupported claims, denominator gaps, resource surprises, processor drift, and safety-critical regressions harder to hide behind one successful video or average.

Done when

A reviewer can reproduce identities and calculations, every unknown and regression is visible, the decision follows unchanged gates, and deployment scope never exceeds the tested surface.

Build today

Benchmark a supported LeRobot/OpenVLA policy, trace its processors and normalized actions, then design or run a LoRA adaptation with a frozen baseline.

Evidence to save

DONE when the weekly ship note explains how “Adaptation report with compute, data, latency, and risk budget” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Publishing only the best checkpoint and strongest metric.

Better

Preserve run lineage, failed checkpoints, baseline, per-condition results, regressions, resource costs, and raw evidence so selection and limitations remain auditable.

Wrong

Estimating missing compute or energy as if it were measured.

Better

Label direct measurements, bounded calculations, assumptions, and unknowns separately; do not convert parameter count into invented power, price, or latency.

Wrong

Approving a model name rather than an artifact and scope.

Better

Authorize one hashed model-processor-adapter bundle for one robot profile, environment, limit set, monitor, expiry, and rollback path; changes trigger declared revalidation.

Job connection

How this becomes employable evidence

Chair a learned-policy release review that reconciles data, compute, task, latency, guard, failure, operator, and rollback evidence into an artifact-specific decision with enforceable scope.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Application / ROS 2 Integration Engineer
Robot HMI / Control & Monitoring Engineer

Chapter 19 interview drill

Interview questions: Adaptation report with compute, data, latency, and risk budget

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A VLA adapter improves aggregate success by 12.5 percentage points but misses one held-out stratum, emits one prohibited action, and exceeds p95 latency. Build the report and defend deploy, revise, or reject.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why does 82.5% overall success not override one prohibited action?

Model interview answer

The prohibited-action threshold is a separate zero-tolerance mandatory gate; an aggregate metric cannot compensate for its failure.

Q2What is the candidate's p95 deadline miss at 125 ms against 100 ms?

Model interview answer

It misses by 25 ms, and the report must also connect that tail delay to stale actions, queueing, fallbacks, and sustained-load behavior.

Q3What exactly receives a deploy decision?

Model interview answer

One identified model, processor, statistics, adapter, runtime, robot profile, environment, limit, monitoring, and rollback bundle—not a model-family name.

Chapter starter artifact

Reject a processor contract that silently swaps cameras

Publish one Policy Passport that identifies the frozen baseline and candidate, proves processor and action round-trips on golden fixtures, separates measured system parameters from declared randomization ranges, reports the predeclared generalization matrix, and ends with an evidence-backed deploy, revise, or reject decision inside an explicit sim-to-real boundary.

week-19-policy-passport.mjsJavaScriptDownload starter

const requiredCameras = ["front", "wrist"];
const limits = { x: 0.04, y: 0.04, gripper: 1 };
const requiredActionFields = Object.keys(limits).sort();
const baseline = { cameras: ["front", "wrist"], action: { x: 0.02, y: -0.01, gripper: 1 } };
const plantedFailure = { cameras: ["wrist", "front"], action: { x: 0.02, y: -0.01, gripper: 1 } };
const missingAction = { cameras: ["front", "wrist"], action: { x: 0.02, y: -0.01 } };

function review(candidate) {
  if (!Array.isArray(candidate?.cameras) ||
      candidate.cameras.length !== requiredCameras.length ||
      candidate.cameras.some((camera, index) => camera !== requiredCameras[index])) {
    return "REJECT camera_order";
  }
  const fields = Object.keys(candidate.action ?? {}).sort();
  if (fields.length !== requiredActionFields.length ||
      fields.some((field, index) => field !== requiredActionFields[index])) {
    return "REJECT action_schema";
  }
  for (const field of requiredActionFields) {
    const value = candidate.action[field];
    if (!Number.isFinite(value) || Math.abs(value) > limits[field]) {
      return "REJECT action_limit";
    }
  }
  return "ACCEPT";
}

console.log("baseline contract: " + review(baseline));
console.log("planted camera-order swap: " + review(plantedFailure));
console.log("missing action field: " + review(missingAction));
console.log("release decision: HOLD");

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-19-policy-passport.mjs
Expected output: baseline contract: ACCEPT planted camera-order swap: REJECT camera_order missing action field: REJECT action_schema release decision: HOLD
Planted failure to diagnose: The processor swaps camera order while preserving shapes, and a second candidate omits one required action field; both fail before any release decision.