Chapter 09 · Turn camera pixels into measured, debuggable robot observations
Today in the field story
One problem, then the next
Introduce the Blue-Crate Inspection Cell’s pre-trained detector as a versioned box-and-score proposal service, not an oracle. Freeze model identity, color order, resize, normalization, backend, NMS, labels, and one-to-one matching rule. Sweep confidence on validation data, keep TP, FP, and FN overlays, then lock the chosen threshold before held-out evaluation and safe rejection behavior.
- Why now
The cell needs object observations beyond fiducials, with task costs and errors made explicit.
- Ignore today
Ignore how the network is trained; Chapter 13 will open that model-engineering layer.
- Unlocks next
Timestamped crate observations that a tracker can associate across frames.
Understand
Build the physical picture first
An object detector proposes labelled rectangles with scores; a frozen matching rule and task cost decide which proposals are useful, missed, or dangerous.
An object detector normally returns candidate class labels, image regions such as bounding boxes, and numeric scores. The score's exact meaning depends on the model and post-processing; it is not automatically a calibrated probability that the physical object exists. Before inference, color order, resize or letterbox behavior, input size, numeric scaling, and normalization must match the model contract. A pipeline can produce confident nonsense when one preprocessing step differs from training.
Evaluation compares predictions with labels under a declared matching rule. A common rule requires the class to match and bounding-box intersection over union, IoU, to reach a chosen threshold. Each prediction and label may be matched at most once. A matched prediction is a true positive, an unmatched prediction is a false positive, and an unmatched label is a false negative. Non-maximum suppression removes selected overlapping candidates, but its score and IoU thresholds are additional configuration, not proof that the remaining box is correct.
A confidence threshold decides which scored predictions enter the task. With raw outputs and other settings fixed, raising the minimum keeps fewer or the same candidates; false positives may fall, but real low-scored objects may become false negatives. Precision asks what fraction of reported objects were correct, while recall asks what fraction of labelled objects were found. Choose a threshold on representative validation data using the real cost of a false grasp, missed package, unnecessary stop, or human review—not the prettiest percentage.
Detection is still an observation boundary. Preserve the image timestamp, inference completion time, model and label-map versions, preprocessing settings, confidence and NMS thresholds, and rejection reason. A box does not provide trustworthy depth, 3D pose, grasp clearance, or permission to move. Downstream code must check freshness and required geometry, and it must represent no accepted detection as an explicit absence of evidence rather than coordinates filled with zeros.
Words you need
Name each idea precisely
- Bounding box
A rectangular image region proposed or labelled around an object.
Physical example:Four pixel coordinates enclose a red block in a tabletop image.
- Confidence score
A model-specific ranking value associated with a predicted class or object candidate.
Physical example:Two cup candidates receive scores 0.82 and 0.47, but neither score by itself proves a cup exists.
- Intersection over union
Overlap area divided by combined union area for two image regions.
Physical example:A predicted box overlapping 60 of 100 union pixels with a label has IoU = 0.60.
- Non-maximum suppression
Post-processing that keeps selected high-scored boxes and removes sufficiently overlapping alternatives.
Physical example:Three boxes around one block are reduced to one candidate under the configured score and IoU rules.
- Precision
The fraction TP/(TP+FP) of accepted predictions that match labels under the frozen rule.
Physical example:Eight correct boxes and two false boxes give precision 8/10 = 80%.
- Recall
The fraction TP/(TP+FN) of labelled objects that receive accepted matching predictions.
Physical example:Eight found blocks and four missed blocks give recall 8/12 ≈ 66.7%.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalPixels, camera projection, and calibration error
A pixel becomes useful only after camera geometry and uncertainty are known.
- u, v
- pixel column and rowUnit: pixels (px)
- fₓ, fᵧ
- camera focal scaleUnit: pixels (px)
- Z
- depth along the camera axisUnit: metres (m)
Use x = (u − cₓ)Z/fₓ. Let u − cₓ = 100 px, Z = 2 m, fₓ = 500 px.
Multiply the numerator: 100 × 2 = 200 px·m.
Divide: x = 200/500 = 0.4 m; pixel units cancel, leaving metres.
Mobile camera pixels are familiar; robotics adds calibrated rays, a camera frame, and physical depth.
If u − cₓ = 50 px, Z = 1 m, and fₓ = 500 px, what is x?
x = 50×1/500 = 0.1 m.
At threshold , with , , and ,
Raising the threshold may improve precision and reduce recall, but neither metric measures pose or safety.
Choose between two thresholds with declared error costs
On one frozen validation set, threshold 0.60 gives TP = 8, FP = 2, FN = 4. Threshold 0.80 gives TP = 6, FP = 0, FN = 6. For this supervised tabletop task only, assign cost 5 to a false positive and cost 1 to a false negative.
At 0.60, calculate precision = 8/(8+2) = 80% and recall = 8/(8+4) ≈ 66.7%.
At 0.80, calculate precision = 6/(6+0) = 100% and recall = 6/(6+6) = 50%.
Calculate declared error cost at 0.60: 5(2 false positives) + 1(4 false negatives) = 14.
Calculate declared error cost at 0.80: 5(0 false positives) + 1(6 false negatives) = 6.
Under only this cost model and dataset, prefer 0.80, while requiring a human-review or safe-no-action path for its six misses.
Inspect every changed prediction and repeat on held-out lighting and object-size slices; the arithmetic does not prove the costs, labels, or dataset represent deployment.
Threshold 0.80 has lower declared error cost 6 versus 14, but also lower recall, so its safe missed-object behavior remains part of acceptance.
Threshold choice is a documented task decision over labelled evidence, not a universal preference for the highest score, precision, or recall.
Physical examples
Where this appears in real life
Colored block and bright reflection
A detector proposes one strong box around a paper block and a weaker box around a reflection with a similar color and outline.
Sweep the confidence threshold and keep the reflection image in the validation set; never delete a difficult false positive after seeing the result.
Sorter with different error costs
A tabletop sorter may safely ask a human about a missed object, while a false positive could send a gripper toward empty or obstructed space.
State the consequence and guard for each error type before selecting the threshold; high recall alone is not the task objective.
Hands-on exercise
Make the idea observable
Use a small pretrained detector or an existing project model, 20–40 non-sensitive tabletop images, a fixed label set, and an OpenCV inference script. Keep the images stationary and command no robot.
Freeze a validation manifest with image IDs, labels, class names, boxes, lighting and object-size tags, and a written one-to-one IoU matching rule.
Record model file hash or revision, label map, input size, BGR/RGB conversion, resize or letterbox policy, normalization, backend, confidence threshold, and NMS settings.
Run exactly the same raw model outputs through at least three confidence thresholds while leaving images, labels, preprocessing, matching, and NMS configuration otherwise fixed.
For each threshold, reconcile TP, FP, and FN with annotated overlays, then calculate precision and recall with their numerator and denominator counts.
Create separate contact sheets for correct detections, false positives, and false negatives, including score, IoU, image tag, and source timestamp.
Measure repeated end-to-end time from captured frame availability through final accepted boxes after warm-up, not only the neural-network forward call.
Select or reject a threshold using declared false-positive and false-negative consequences, then lock it before running a separate held-out check.
Increasing the score threshold can remove reflections and also remove real small or shadowed objects. Preprocessing or NMS changes can alter results even when the displayed confidence threshold stays fixed.
The report can reproduce each threshold row, trace every count to an overlay, state the chosen task cost and safe miss behavior, and show held-out accuracy plus end-to-end timing.
Build today
Detect, track, and estimate the pose of tabletop objects with an annotated evaluation set.
Evidence to save
DONE when a comparison table for “Object detection and confidence thresholds” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Interpreting a confidence score of 0.90 as a guaranteed 90% chance that the physical object is present.
Treat scores according to the specific model contract and measure their behavior on representative labelled data; calibration and real-world validity require evidence.
Selecting the threshold with the highest precision without examining missed objects.
Report precision and recall with counts, inspect false negatives, and use explicit task consequences plus a safe missed-object path.
Tuning thresholds on a set and reporting the same set as unbiased final evidence.
Use a validation set for selection, lock the full configuration, and report results on separate held-out examples and important deployment slices.
Changing model preprocessing while calling the experiment a threshold-only comparison.
Freeze color order, resize, numeric scale, normalization, backend, and NMS; change only the confidence threshold for that comparison.
Job connection
How this becomes employable evidence
For a deployed perception release, the engineer freezes preprocessing, model and label versions, IoU matching, NMS, and a confidence sweep, then links every threshold decision to false-grasp versus missed-object cost and verifies a no-action fallback.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robot Learning Deployment / Physical AI Integration Engineer
Chapter 09 interview drill
Interview questions: Object detection and confidence thresholds
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A detector has 100% precision at threshold 0.9 and 70% precision at 0.5. Explain why that does not select 0.9, then define labels, IoU matching, TP/FP/FN, NMS, task costs, held-out slices, and freshness checks.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Under a one-to-one IoU matching rule, what are TP, FP, and FN?
A TP is one accepted prediction matched to one label; an unmatched accepted prediction is an FP; an unmatched label is an FN.
Q2Why can raising a confidence threshold improve precision while reducing recall?
It removes low-scored candidates that may include false positives and real objects, so accepted predictions can be cleaner while more labels are missed.
Q3What must be frozen for a fair confidence-threshold sweep?
Images, labels, matching rule, preprocessing, model and label versions, backend, NMS, timing method, and every setting except the threshold being compared.