Chapter 09 · Turn camera pixels into measured, debuggable robot observations
Today in the field story
One problem, then the next
The Blue-Crate Inspection Cell must now turn image points into rays. Measure the calibration target, capture varied sharp views, estimate intrinsics and distortion, and reserve held-out positions near the image edges. Report per-view and spatial reprojection residuals, then resize one frame deliberately to prove that unchanged lens hardware does not make old pixel geometry valid.
- Why now
Pose and size claims require a calibrated camera model rather than a convincing overlay.
- Ignore today
Ignore camera-to-robot extrinsics and 3D depth until Chapter 10.
- Unlocks next
A qualified pixel-to-ray model for fiducial and object geometry.
Understand
Build the physical picture first
Camera calibration is a measured dictionary between 3D rays and 2D pixels, plus a correction for how the real lens bends that ideal geometry.
The pinhole model begins in the camera frame, where a point has coordinates X, Y, Z and positive Z lies in front of the camera under the usual OpenCV calibration convention. Ignoring distortion, its pixel is u = f_x X/Z + c_x and v = f_y Y/Z + c_y. The focal lengths f_x and f_y are measured in pixels; c_x and c_y locate the principal point. Dividing by Z explains why the same object appears smaller when it is farther away.
The intrinsic matrix K stores f_x, f_y, c_x, and c_y for one image geometry. Real lenses also introduce radial distortion, which bends points increasingly with radius, and tangential distortion, which models lens-to-sensor misalignment. OpenCV stores fitted distortion coefficients beside K. Calibration parameters cannot be copied blindly after changing resolution, digital crop, lens, focus setting, or camera module; either transform the intrinsics under a known image resize/crop or recalibrate and validate the new path.
Calibration needs known 3D target points paired with detected 2D image points. A flat checkerboard supplies repeated corners with a measured spacing, but useful views must cover the image, vary tilt and position, and provide enough geometric diversity. Twenty nearly identical front-facing images repeat almost the same constraint. Blurred corners, a bent print, an unmeasured square size, or a board seen only near the center can produce plausible numbers that fail where the robot actually looks.
OpenCV calibration estimates K, distortion, and a rotation/translation for the target in each calibration view. Those per-view poses are not the fixed transform from the camera housing to the robot base. Reprojection error compares observed corners with corners projected by the fitted model; inspect per-view and spatial residuals, not only one average. Finally, reserve target views or measured points that were not used for fitting so validation can reveal overfitting, edge bias, or a changed camera configuration.
Words you need
Name each idea precisely
- Camera intrinsics
Parameters inside the camera model that map camera-frame rays to pixel coordinates.
Physical example:The stored f_x, f_y, c_x, and c_y describe one camera and image geometry.
- Focal length in pixels
The scale that converts a camera-frame direction ratio such as X/Z into pixel offset.
Physical example:With f_x = 500 px, a ratio X/Z = 0.1 produces a 50 px horizontal offset.
- Principal point
The pixel location c_x, c_y where the ideal optical axis meets the image plane.
Physical example:A principal point near (320, 240) lies near the center of a 640 × 480 image.
- Distortion coefficients
Fitted values that describe selected radial and tangential departures from ideal pinhole projection.
Physical example:Straight shelf edges bow outward near the corners until the lens distortion is corrected.
- Reprojection residual
The pixel difference between an observed target point and the point predicted by the fitted camera model.
Physical example:A detected corner at u = 372 and a projected corner at u = 370 have a 2 px horizontal residual.
- Calibration-view pose
The estimated rotation and translation that place the known target relative to the camera for one image.
Physical example:Tilting the checkerboard creates a new per-image pose while the camera intrinsics stay fixed.
Visual model
See the relationship
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalPixels, camera projection, and calibration error
A pixel becomes useful only after camera geometry and uncertainty are known.
- u, v
- pixel column and rowUnit: pixels (px)
- fₓ, fᵧ
- camera focal scaleUnit: pixels (px)
- Z
- depth along the camera axisUnit: metres (m)
Use x = (u − cₓ)Z/fₓ. Let u − cₓ = 100 px, Z = 2 m, fₓ = 500 px.
Multiply the numerator: 100 × 2 = 200 px·m.
Divide: x = 200/500 = 0.4 m; pixel units cancel, leaving metres.
Mobile camera pixels are familiar; robotics adds calibrated rays, a camera frame, and physical depth.
If u − cₓ = 50 px, Z = 1 m, and fₓ = 500 px, what is x?
x = 50×1/500 = 0.1 m.
The pinhole projection is
Precision and recall can score target detection, but calibration needs a quantity such as the stated reprojection error.
Project one camera-frame point and inspect a residual
Ignore distortion for the first calculation. A calibrated image has f_x = f_y = 500 px, c_x = 320 px, c_y = 240 px. A target point in the camera frame is X = 0.10 m, Y = 0.05 m, Z = 1.00 m, and its observed corner is (u, v) = (372, 264).
Check units and visibility: X, Y, and Z use metres, focal lengths use pixels, and Z = 1.00 m is positive.
Calculate u = f_x X/Z + c_x = 500(0.10/1.00) + 320 = 370 px.
Calculate v = f_y Y/Z + c_y = 500(0.05/1.00) + 240 = 265 px.
Form the predicted pixel (370, 265) and subtract it from the observed pixel (372, 264) to get residual components (+2, -1) px.
Calculate residual magnitude √(2² + (-1)²) = √5 ≈ 2.24 px for this held-out point.
Repeat the projection across held-out center and edge points before deciding whether about 2.24 px is acceptable for the physical task; one residual does not validate the whole camera.
The ideal model projects the point to (370, 265), while the held-out observation differs by about 2.24 px.
Projection turns declared 3D geometry into a testable pixel prediction; residuals show where the fitted model and observed image disagree.
Physical examples
Where this appears in real life
Checkerboard across the whole frame
A measured flat checkerboard is photographed near the center, near all four corners, at several tilts, and at more than one distance.
Corner coverage and pose diversity constrain different parts of the model; reject blur and partial detections instead of increasing image count with poor evidence.
Straight shelf near a wide-angle lens edge
A physically straight shelf edge appears curved near the outside of a wide-angle image but closer to straight after undistortion.
Check held-out line and target points across the frame. A pleasing center crop does not validate corner geometry or metric projection.
Hands-on exercise
Make the idea observable
Use one fixed-focus camera, a flat checkerboard or ChArUco target with precisely measured spacing, OpenCV calibration tools, and a stationary workspace. Keep powered robot motion disabled.
Measure the printed target spacing and one full grid width with a ruler or caliper, record the units, and reject a curled or stretched print.
Capture at least 15 sharp, complete target views that cover the center, edges, and corners with varied tilt, position, and distance while keeping camera resolution and focus unchanged.
Review detected corners overlaid on every image; remove blurred, partial, or wrongly ordered detections and record each rejection reason.
Reserve several geometrically varied images for validation, then run calibration only on the remaining fit set and save K, distortion coefficients, image size, model flags, and software version.
Project fitted target points back into every fit and held-out image, calculate per-view and per-corner residuals, and plot or tabulate residual magnitude by image region.
Undistort a held-out image and inspect straight features plus target corners near the edges; do not use visual straightness as a replacement for numeric residuals.
Change one camera condition such as digital resize after saving the baseline, prove the old unmodified intrinsics no longer match that image geometry, and restore the qualified configuration.
Varied sharp views usually constrain the model better than repeated center views. A low fit average can coexist with large held-out or edge residuals, especially after image geometry changes.
The versioned calibration includes measured target geometry, qualified image settings, accepted and rejected view lists, K and distortion, fit and held-out residuals by region, and a stated task tolerance.
Build today
Detect, track, and estimate the pose of tabletop objects with an annotated evaluation set.
Evidence to save
DONE when a deterministic “Camera intrinsics, distortion, and calibration” failure test reports expected versus actual behavior and passes after the documented fix.
Common mistakes
Catch the wrong mental model
Collecting many nearly identical front-facing checkerboard images and calling the large image count diverse.
Cover the image plane and vary target tilt, position, and distance while keeping corners sharp and the target physically flat.
Accepting one low average reprojection error without inspecting held-out views or edge residuals.
Report per-view and spatial residual distributions on both fit and reserved data, then compare them with a physical task tolerance.
Using each checkerboard view's fitted rotation and translation as the fixed camera-to-robot mounting transform.
Treat those values as target-to-camera pose for that image; estimate and validate the separate sensor-to-robot extrinsic transform with its own procedure.
Reusing K unchanged after cropping or resizing the image because the lens did not move.
Qualify the complete image geometry; transform intrinsics correctly for a known resize/crop or recalibrate and validate the new configuration.
Job connection
How this becomes employable evidence
Before a vision-guided robot trial, the engineer qualifies the exact camera mode, measures the target, estimates intrinsics and distortion, rejects weak views, checks held-out residuals across the frame, and separately verifies the camera-to-robot transform.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 09 interview drill
Interview questions: Camera intrinsics, distortion, and calibration
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A camera reports 0.4 px average reprojection error, yet picks are wrong near the image edge. Explain the pinhole model, distortion, view diversity, held-out residual map, resize or focus risks, and why calibration-view poses are not robot extrinsics.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1In u = f_x X/Z + c_x, why does horizontal pixel offset shrink when Z grows and X stays fixed?
The direction ratio X/Z becomes smaller, so multiplying by the same pixel focal length produces a smaller offset from the principal point.
Q2Why are many centered, front-facing checkerboard views weak calibration evidence?
They repeat similar constraints and provide little information about distortion and projection behavior at other tilts, depths, and image regions.
Q3What does a low reprojection error not prove?
It does not prove held-out accuracy, correct target measurements, unchanged camera settings, correct camera-to-robot extrinsics, or acceptable physical task error.