Chapter 09 · Turn camera pixels into measured, debuggable robot observations

Today in the field story

One problem, then the next

Seal the Blue-Crate Inspection Cell evaluation before the final run: dataset manifest, labels, matching rule, model and pipeline configuration, threshold, timing boundary, hardware, slices, and gates. Recalculate precision and recall with counts, report median, p95, maximum latency, and retain every failed overlay. A passing observation contract remains separate from downstream motion and safety approval.

Why now: The chapter needs a verdict that another engineer can reproduce and challenge from item-level evidence.
Ignore today: Ignore 3D target accuracy and closed-loop behavior until the next chapters.
Unlocks next: A frozen vision baseline ready for depth geometry and robot-shaped datasets.

Understand

Build the physical picture first

A vision evaluation is a sealed measuring course: inputs, labels, rules, configuration, hardware, and timing stay fixed so results can be repeated and challenged.

A defensible test starts with a protocol, not a metric copied after the run. Freeze the image or video manifest, label revision, class definitions, box or identity matching rule, thresholds, model and preprocessing versions, camera conditions, compute hardware, warm-up rule, and exact timing boundary. The same files must produce the same scored decisions. If a threshold changes after failures are seen, create a new version and rerun rather than silently replacing the original result.

Precision and recall need their TP, FP, and FN counts because percentages can hide a tiny sample or different denominators. Report results by meaningful slices such as bright versus dim light, small versus large object, center versus edge, clear versus occluded, and stationary versus moving. Aggregate results can pass while one physically important slice fails. Contact sheets should include ordinary successes, false positives, false negatives, poor localization, ID switches, and rejected frames—not only attractive examples.

Latency must match the robot decision boundary. Neural-network forward time excludes capture, decoding, resize, queueing, post-processing, and message transport, so measure capture-available to final-observation time when that is what the consumer experiences. Discard or label warm-up separately, repeat enough samples, and report a distribution such as median, 95th percentile, and maximum with hardware and load. An 85 ms 95th percentile fails a 70 ms deadline even when average inference is fast.

Acceptance combines accuracy, timing, geometry, freshness, and failure behavior without pretending that any one metric proves safety. Publish a manifest, runnable command, environment and configuration, summary table, slice table, overlays, contact sheets, and the largest known limitation. Keep validation data separate from threshold selection, prevent private images from entering artifacts, and make no-action or human-review behavior part of the test whenever observations are missing, stale, or outside tolerance.

Words you need

Name each idea precisely

Frozen test protocol: A versioned definition of inputs, labels, matching, configuration, hardware, and measurements that cannot change within one reported run.
Physical example:
A manifest names 60 tabletop images, label revision 3, IoU 0.5, model hash, threshold 0.7, and one edge computer.
Validation set: Labelled data used to choose settings or compare candidate approaches before final held-out evaluation.
Physical example:
Thirty lighting-varied images are used to select a confidence threshold, so they are not presented as untouched final proof.
Dataset slice: A declared subset sharing a condition that may reveal a concentrated failure.
Physical example:
Objects smaller than 40 pixels wide form a small-object slice with its own counts.
Latency percentile: A time below which a stated percentage of measured samples completed.
Physical example:
A p95 of 85 ms means 95% of measured observations finished in 85 ms or less under that run condition.
Failure taxonomy: A stable set of categories used to classify distinct ways the pipeline fails.
Physical example:
Glare miss, small-object miss, wrong class, poor box, ID switch, stale result, and timeout are separate categories.
Held-out evidence: Examples not used to tune the reported model, threshold, or other configuration.
Physical example:
A separate sealed tabletop sequence is scored only after the confidence and NMS settings are locked.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalPixels, camera projection, and calibration error

A pixel becomes useful only after camera geometry and uncertainty are known.

u, v: pixel column and rowUnit: pixels (px)
fₓ, fᵧ: camera focal scaleUnit: pixels (px)
Z: depth along the camera axisUnit: metres (m)

Use x = (u − cₓ)Z/fₓ. Let u − cₓ = 100 px, Z = 2 m, fₓ = 500 px.
Multiply the numerator: 100 × 2 = 200 px·m.
Divide: x = 200/500 = 0.4 m; pixel units cancel, leaving metres.

Programmer analogy

Mobile camera pixels are familiar; robotics adds calibrated rays, a camera frame, and physical depth.

If u − cₓ = 50 px, Z = 1 m, and fₓ = 500 px, what is x?

x = 50×1/500 = 0.1 m.

With $TP=45$ , $FP=5$ , and $FN=10$ ,

\mathrm{precision}=\frac{45}{50}=90\%,\qquad \mathrm{recall}=\frac{45}{55}\approx81.8\%.

A $95$ th-percentile latency of $85\,\mathrm{ms}$ exceeds the $70\,\mathrm{ms}$ deadline, so the detector still fails its timing requirement.

Decide an accuracy-and-latency acceptance gate

A locked held-out run yields TP = 45, FP = 5, FN = 10. The protocol requires precision at least 88%, recall at least 80%, and capture-to-final-observation p95 latency no more than 70 ms. The measured p95 is 85 ms.

Calculate prediction count TP+FP = 45+5 = 50 and precision = 45/50 = 90%.
Calculate labelled-object count TP+FN = 45+10 = 55 and recall = 45/55 ≈ 81.8%.
Compare 90% precision with the 88% minimum and mark that accuracy condition passed.
Compare about 81.8% recall with the 80% minimum and mark that accuracy condition passed.
Compare p95 latency 85 ms with the 70 ms maximum: 85 - 70 = 15 ms over budget, so timing fails.
Return the overall gate as failed and inspect slow-case traces; two passing accuracy metrics cannot cancel one required timing failure.

Result

Precision and recall pass their thresholds, but p95 latency exceeds the deadline by 15 ms, so the complete acceptance gate fails.

What this proves

A robot observation is acceptable only when every required accuracy, timing, geometry, and validity condition passes under the same frozen run.

Physical examples

Where this appears in real life

Bin-pick inspection under two lights

The same set of paper parts is inspected under bright diffuse light and a dim side light, with identical labels and locked model settings.

Look for:

Report each lighting slice separately; a strong bright-light aggregate must not hide dim-light misses or increased latency from camera exposure.

Conveyor timing under normal load

Recorded package frames are replayed while the edge computer also runs the normal logging and communication workload.

Look for:

Measure capture-to-final-observation latency after warm-up and preserve slow cases, instead of quoting isolated forward-pass time on an idle machine.

Hands-on exercise

Make the idea observable

Use the Week 09 saved images or videos, labels, calibrated camera configuration, detector or classical baseline, tracking output where available, and one declared computer. Do not add new examples after seeing results.

Create a read-only test manifest with file identifiers, label revision, train/validation/held-out membership, lighting, size, occlusion, motion, and image-region tags.
Write the matching, identity, calibration-residual, rejection, and timing rules plus accuracy and latency gates before running the held-out set.
Lock camera/image settings, code revision, dependencies, model hash, preprocessing, confidence, NMS, tracker lifecycle, hardware, workload, and one runnable command.
Run warm-up separately, then score the held-out manifest once while saving per-item TP/FP/FN or tracking events, geometry residuals, rejection reasons, and capture-to-final timestamps.
Calculate totals and slice tables with raw denominators, then report median, p95, and maximum end-to-end latency and link slow samples to traces.
Build compact contact sheets for successes and every failure class, including scores, residuals, age, latency, and image IDs without private content.
Issue PASS only if every predeclared gate passes; otherwise issue NEEDS REVISION with the first causal failure stage, retained baseline, and one bounded next experiment.

Observe

Aggregate accuracy can hide a weak slice, and average inference time can hide deadline misses. Frozen settings make a failure reproducible instead of negotiable after the fact.

Done when

Another engineer can run one command against the same manifest, reproduce totals and slice counts, trace every metric to item evidence, and reach the same PASS or NEEDS REVISION decision.

Build today

Detect, track, and estimate the pose of tabletop objects with an annotated evaluation set.

Evidence to save

DONE when the weekly ship note explains how “Vision evaluation: precision, recall, latency, and examples” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Showing only successful overlays or one aggregate percentage.

Better

Publish TP/FP/FN counts, meaningful slice tables, and contact sheets for every failure category as well as representative successes.

Wrong

Reporting average neural-network forward time as the robot's observation latency.

Better

Measure the consumer-relevant capture-to-final-observation boundary and report median, p95, maximum, hardware, workload, and warm-up treatment.

Wrong

Changing a threshold after inspecting held-out failures and keeping the old test label.

Better

Version the change, treat those examples as seen, lock a new protocol, and evaluate on genuinely held-out evidence.

Wrong

Declaring the pipeline safe because accuracy and latency gates passed.

Better

State that the test supports only its declared observation contract; separate safety analysis, downstream guards, hardware behavior, and broader operating conditions still require evidence.

Job connection

How this becomes employable evidence

Before releasing a perception update, the engineer freezes the dataset, labels, configuration, hardware, matching and timing rules, then reports accuracy slices, calibration residuals, p95 latency, contact sheets, and safe rejection behavior with a reproducible command.

Relevant target roles

Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robot Learning Deployment / Physical AI Integration Engineer

Chapter 09 interview drill

Interview questions: Vision evaluation: precision, recall, latency, and examples

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A vision demo shows 90% precision and a fast average time. Design a release gate that exposes denominators, recall, data leakage, weak slices, calibration error, capture-to-result p95, stale outputs, reproducibility, and safe failure behavior.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why should precision and recall be reported with TP, FP, and FN counts?

Model interview answer

The counts expose denominators and evidence size, allow recalculation, and show whether false positives or misses are driving each percentage.

Q2Why can average inference time pass while the deployed vision path misses its deadline?

Model interview answer

The average can hide a slow tail and may exclude capture, decoding, queueing, preprocessing, post-processing, and transport that the robot actually waits for.

Q3What makes held-out evidence different from validation data?

Model interview answer

Validation data influenced model or setting choices; held-out evidence remains unseen by those choices until the full configuration and protocol are locked.

Chapter starter artifact

Score a frozen crate-detection threshold

A frozen tabletop dataset reproduces image metadata, classical intermediate stages, held-out calibration residuals, fiducial identity and pose checks, detector threshold counts, tracking identity errors, p95 end-to-end latency, bad-case slices, and a scoped PASS or NEEDS REVISION verdict.

week-09-blue-crate.mjsJavaScriptDownload starter

const mission = "blue-crate";
const threshold = 0.7;
const latencyLimitMs = 100;
const cases = [
  { truth: true, score: 0.95, latencyMs: 60 },
  { truth: true, score: 0.80, latencyMs: 72 },
  { truth: true, score: 0.40, latencyMs: 65 },
  { truth: true, score: 0.75, latencyMs: 110 },
  { truth: false, score: 0.85, latencyMs: 70 },
];
const validCases = cases.every((item) =>
  typeof item.truth === "boolean" &&
  Number.isFinite(item.score) && item.score >= 0 && item.score <= 1 &&
  Number.isFinite(item.latencyMs) && item.latencyMs >= 0
);
if (!validCases) throw new Error("invalid evaluation case");
const accepted = cases.map((item) => item.score >= threshold);
const tp = cases.filter((item, index) => item.truth && accepted[index]).length;
const fp = cases.filter((item, index) => !item.truth && accepted[index]).length;
const fn = cases.filter((item, index) => item.truth && !accepted[index]).length;
const precision = tp / (tp + fp);
const recall = tp / (tp + fn);
const latencyGate = cases.reduce((worst, item) => Math.max(worst, item.latencyMs), 0) <= latencyLimitMs;
const status = precision >= 0.75 && recall >= 0.75 && latencyGate ? "PASS" : "NEEDS_REVISION";
console.log("mission=" + mission);
console.log("tp=" + tp + " fp=" + fp + " fn=" + fn);
console.log("precision=" + precision.toFixed(3));
console.log("recall=" + recall.toFixed(3));
console.log("latencyGate=" + (latencyGate ? "PASS" : "FAIL"));
console.log("status=" + status);

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-09-blue-crate.mjs
Expected output: mission=blue-crate tp=3 fp=1 fn=1 precision=0.750 recall=0.750 latencyGate=FAIL status=NEEDS_REVISION
Planted failure to diagnose: Change the final truth label from boolean false to the string false. JavaScript would otherwise treat it as truthy, so the schema guard must reject the corrupted evaluation row.