Chapter 09 · Turn camera pixels into measured, debuggable robot observations
Today in the field story
One problem, then the next
Place a measured fiducial beside the Blue-Crate Inspection Cell as a controlled engineering baseline. Preserve dictionary, printed size, corner order, calibration, frame, and timestamp; score identity separately from pose error. Test glare, clipping, tilt, distance, and wrong scale while retaining rejected candidates so a correct ID cannot hide poor geometry.
- Why now
A known visual target separates camera and geometry problems from learned object-recognition problems.
- Ignore today
Ignore using marker pose as a motion command.
- Unlocks next
A repeatable reference against which the crate detector can be evaluated.
Understand
Build the physical picture first
A fiducial marker is a measured visual signpost: its code supplies identity and its corners supply geometry, but calibration and validation decide whether the pose is trustworthy.
An ArUco marker is a square binary pattern selected from a named dictionary. Detection first finds quadrilateral candidates, then samples their inner cells and checks whether a code belongs to the configured dictionary. Successful output includes an ID and four ordered pixel corners; rejected candidates are useful debug evidence. The ID belongs to the printed code, not automatically to the object beside it, and choosing the wrong dictionary can turn a correct print into repeated misses.
Known marker geometry connects pixels to pose. Given calibrated camera intrinsics, distortion coefficients, ordered corners, and the marker's measured side length, a perspective-n-point calculation can estimate a rotation and translation between the marker frame and camera frame. The translation uses the same physical unit supplied for marker length. A 0.050 m marker entered as 50 without declaring millimetres can produce a pose scale one thousand times too large even though the axis overlay looks orderly.
Fiducials are a strong engineering baseline because the pattern, identity, corner order, and side length are controlled. Before testing a learned detector, a marker can reveal camera encoding errors, weak focus, calibration problems, frame mistakes, pose latency, and distance-dependent accuracy. It cannot prove the learned object detector works, and successful decoding cannot prove the pose is within tolerance. Detection counts and pose residuals answer separate questions.
Print and scene quality still matter. Scaling in a printer dialog changes the physical marker length; curled paper violates the assumed plane; glare hides bits; motion blur moves corners; partial occlusion may prevent decoding; and corners near the image border may be poorly constrained or clipped. Test known IDs at measured distances and tilts, keep rejected-candidate overlays, compare estimated translation with a ruler, and verify axes against a deliberately labelled marker frame.
Words you need
Name each idea precisely
- Fiducial marker
A deliberately designed visual target with known identity and geometry.
Physical example:A measured square ArUco print attached flat to a stationary box.
- Marker dictionary
A named collection of valid binary marker codes and their bit size.
Physical example:DICT_6X6_250 and DICT_4X4_50 interpret different grids and valid IDs.
- Corner refinement
A method that improves detected corner locations beyond the initial coarse quadrilateral estimate.
Physical example:Subpixel refinement adjusts a corner from an integer pixel location to a more precise image coordinate.
- Pose estimate
An inferred position and orientation between named coordinate frames.
Physical example:The marker frame is estimated 0.98 m in front of the camera with a small tilt.
- Rejected candidate
A square-like image region considered during detection but not decoded as a valid configured marker.
Physical example:A dark package label has four sides but its sampled bits do not match the marker dictionary.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalPixels, camera projection, and calibration error
A pixel becomes useful only after camera geometry and uncertainty are known.
- u, v
- pixel column and rowUnit: pixels (px)
- fₓ, fᵧ
- camera focal scaleUnit: pixels (px)
- Z
- depth along the camera axisUnit: metres (m)
Use x = (u − cₓ)Z/fₓ. Let u − cₓ = 100 px, Z = 2 m, fₓ = 500 px.
Multiply the numerator: 100 × 2 = 200 px·m.
Divide: x = 200/500 = 0.4 m; pixel units cancel, leaving metres.
Mobile camera pixels are familiar; robotics adds calibrated rays, a camera frame, and physical depth.
If u − cₓ = 50 px, Z = 1 m, and fₓ = 500 px, what is x?
x = 50×1/500 = 0.1 m.
Translation error along is
With , , and , precision and recall are both . Detection metrics and pose error answer different questions.
Score marker detection and pose error separately
A test contains 20 labelled images where marker ID 17 should be visible. The configured detector correctly returns ID 17 in 19 images, reports one wrong extra marker elsewhere, and misses ID 17 once. At a measured z distance of 1.00 m, one successful pose estimates z = 0.98 m.
Set detection counts from the frozen labels: TP = 19, FP = 1, and FN = 1.
Calculate precision = TP/(TP+FP) = 19/(19+1) = 95%.
Calculate recall = TP/(TP+FN) = 19/(19+1) = 95%.
Calculate the example z translation error separately: |0.98 - 1.00| = 0.02 m = 2 cm.
Do not combine 95% and 2 cm into one score; one measures identity detections over images and the other measures one pose component on a successful detection.
Check more distances, tilts, image regions, and full 3D residuals before accepting the pose path for a robot task.
Marker identity detection has 95% precision and 95% recall on this set, while the selected successful pose has a 2 cm z error.
A marker can be decoded correctly yet localized poorly, so identity, pose, timing, and frame checks need separate acceptance limits.
Physical examples
Where this appears in real life
Marker beside a ruler
A flat 50 mm marker is placed beside a ruler at several measured camera distances while the camera remains fixed.
Compare decoded ID, corner overlay, estimated z translation, and physical distance; separate misses from pose errors on successful detections.
Two labelled storage bins
Two stationary bins carry different marker IDs from the same dictionary, and one marker is partly covered by a paper strip.
Verify identity is attached by configuration to the intended bin, log the occluded marker as a miss or rejected candidate, and never guess its current pose from the other bin.
Hands-on exercise
Make the idea observable
Use the calibrated camera from Day 59, one OpenCV-generated marker from a recorded dictionary, a ruler, a flat backing board, and a stationary tabletop. Disable robot motion.
Generate and print one marker with scaling disabled, then measure its outer side length in metres and record marker ID, dictionary, print file revision, and measured size.
Place the marker flat at three measured distances and three tilts, including positions near the image center and edge, while keeping the qualified camera mode unchanged.
Run marker detection, save detected corners, IDs, rejected candidates, source timestamp, and an annotated image for every trial.
Estimate pose with the measured marker length, K, and distortion; draw the returned frame axes and verify their orientation against axes drawn physically beside the marker.
Calculate translation error against ruler measurements for every valid trial and group misses plus large residuals by distance, tilt, glare, and image region.
Cover one corner, add a glare source, and use the wrong dictionary in separate fault trials; require explicit no-detection or rejection rather than a fabricated pose.
Restore the qualified setup, rerun one baseline trial, and save the configuration plus result table so another person can reproduce it.
Correct dictionary, measured print size, calibrated intrinsics, sharp corners, and visible marker area all affect the result. A stable ID does not guarantee stable translation or orientation.
The evidence contains print provenance, measured side length, all test conditions, detected and rejected overlays, identity counts, pose residuals, fault outcomes, and a repeated baseline after restoration.
Build today
Detect, track, and estimate the pose of tabletop objects with an annotated evaluation set.
Evidence to save
DONE when the integrated “Fiducial markers as a reliable engineering baseline” path is observable, cancelable, and leaves the prior baseline reproducible.
Common mistakes
Catch the wrong mental model
Entering markerLength = 50 for a 50 mm print while the rest of the robot uses metres.
Measure and enter 0.050 m, state the unit at the API boundary, and reject translations outside the physically possible range.
Treating a correct marker ID and tidy axis overlay as proof of accurate pose.
Compare translation and orientation with held-out measured poses across distance, tilt, lighting, and image region.
Printing through a fit-to-page dialog and trusting the nominal digital marker size.
Disable scaling, mount the print flat, measure the actual outer side used by the pose model, and store that measurement with the calibration.
Discarding rejected candidates and failed frames because only successful detections seem useful.
Save rejected overlays and structured failure reasons; they distinguish glare, border clipping, occlusion, dictionary mismatch, and threshold problems.
Job connection
How this becomes employable evidence
At a customer-site camera check, the engineer places a qualified fiducial at measured poses to separate focus, calibration, frame, and latency faults before diagnosing the production detector, then records marker misses and pose residuals instead of showing only an axis overlay.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 09 interview drill
Interview questions: Fiducial markers as a reliable engineering baseline
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A marker ID is correct but its estimated distance is wrong. Walk through print scale, marker units, dictionary, corner quality, camera intrinsics and distortion, solvePnP inputs, frame conventions, timestamps, and held-out physical measurements.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What two outputs does a detected ArUco marker provide before pose estimation?
Its decoded marker ID and the ordered image coordinates of its four corners.
Q2Why does the physical marker side length affect estimated translation?
It sets the real-world scale of the 3D corner model, so translation is returned in the same unit and scales with that declared length.
Q3Why is a fiducial useful before a learned detector?
Its controlled identity and geometry provide a repeatable baseline that can isolate camera, calibration, frame, pose, and timing faults from model-recognition faults.