Chapter 10 · Turn depth measurements into safe, inspectable 3D targets
Today in the field story
One problem, then the next
The candidate approach looks correct in the camera frame but misses the tote after conversion to the robot base. Treat the fixed camera extrinsic as a measured transform with a revision, not a diagram arrow. Evaluate held-out points and inspect residual direction as well as magnitude. If the camera moves with the wrist, stop and identify the separate hand–eye problem; this fixed-camera exercise does not solve or validate it.
- Why now
Every downstream pose inherits any camera-to-robot transform error.
- Ignore today
Ignore online self-calibration; establish and challenge one fixed extrinsic with held-out evidence.
- Unlocks next
A versioned transform whose error budget can accompany each target.
Understand
Build the physical picture first
Extrinsic calibration measures how two rigid rulers are bolted together; it must work on new points, not only the poses used to fit it.
Camera intrinsics describe rays inside the camera model. Extrinsics describe the rigid rotation and translation between the camera frame and another frame such as base_link or a tool flange. A fixed room camera is often calibrated to the robot base. A wrist camera also needs the changing robot kinematic chain plus the fixed camera-to-tool transform. Name the transform direction before solving because camera-to-base and base-to-camera are inverses, not interchangeable labels.
A calibration dataset needs known correspondences or target poses observed from varied positions and orientations. Near-identical front-facing views poorly constrain some rotation and translation directions. The target dimensions must use the declared physical unit, detections must be accurate, timestamps must match the robot pose, and the camera mount must remain rigid. If the lens focus, image resolution, mount, or robot geometry changes, decide whether the calibration is still valid instead of silently reusing it.
The estimated transform should be a rigid transform: its rotation is orthonormal with determinant near +1, and translation is expressed in the chosen physical unit. Fitting error on the calibration observations is useful but can hide overfitting, bad coverage, or a shared measurement bias. Reserve held-out target poses and known 3D checkpoints. Transform their camera observations into the robot frame and report per-axis and Euclidean error across near, far, centre, edge, high, and low regions.
A visually aligned overlay can still be unsafe. At long range, a small rotation error produces a larger lateral position error. On a moving wrist, a timestamp offset looks like a spatial calibration error because the camera pose changed during the offset. Version the intrinsics, extrinsic transform, mount state, robot description, solver settings, dataset, residuals, validation results, and validity conditions so field diagnosis can separate a loose bracket from code or timing.
Words you need
Name each idea precisely
- Intrinsic calibration
Camera parameters such as focal lengths, principal point, and distortion that connect camera rays to image pixels.
Physical example:Changing the image resolution may require scaled focal lengths and principal point even though the camera remains mounted in the same place.
- Extrinsic calibration
The rigid rotation and translation relating a sensor frame to another physical frame.
Physical example:A fixed camera may be measured as 0.35 m above and rotated relative to the robot base.
- Eye-in-hand
A camera mounted on the moving robot tool or wrist, with a fixed camera-to-tool relationship.
Physical example:The camera moves whenever the arm moves, while its bolted offset from the wrist should stay constant.
- Held-out validation
Testing calibration on observations that were not used to estimate its parameters.
Physical example:Five target poses are reserved until after fitting and reveal large error near the edge of the workspace.
- Euclidean position error
The straight-line distance between an estimated 3D point and its trusted reference.
Physical example:Coordinate errors of 0.03 m and 0.04 m on two axes combine into 0.05 m straight-line error.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalDepth, point distance, and rigid alignment
3D perception needs distances and poses that agree across frames.
- p = [x,y,z]
- a 3D pointUnit: metres (m)
- ||p||
- distance from the frame originUnit: metres (m)
- eᵢ
- alignment residual for point iUnit: metres (m)
For p = [0.3, 0.4, 0] m, square the components: 0.09 + 0.16 + 0.
The sum is 0.25 m².
Distance is √0.25 = 0.5 m; invalid or missing depth must be excluded before this calculation.
Like mapping API objects between schemas, point clouds require an explicit transform; unlike JSON, a wrong transform can still look plausible.
How far is [0, 0, 2] m from the camera origin?
2 m.
For aligned axes,
where the camera coordinate can come from . This translation-only check does not cover rotation or calibration uncertainty.
Validate a simple aligned-axis extrinsic on an unseen point
For a teaching case with aligned axes, calibration estimates the camera origin at translation (0.20, -0.05, 0.30) m in base_link. A held-out point is measured at (0.40, 0.10, 0.50) m in the camera frame. Its surveyed base-frame reference is (0.61, 0.04, 0.81) m.
Confirm the transform direction maps camera coordinates into
base_link; this example deliberately has identity rotation.Transform the held-out point by adding translation: estimate
(0.60, 0.05, 0.80) minbase_link.Compute coordinate errors against the reference:
(-0.01, 0.01, -0.01) m.Compute Euclidean error:
sqrt(0.01² + 0.01² + 0.01²) ≈ 0.0173 m, or 17.3 mm.Compare with the frozen acceptance limit; if the requirement is at most 20 mm, this held-out point passes.
Keep the per-point result and test the other reserved locations because one passing centre point cannot validate rotation, workspace coverage, or timing.
The unseen checkpoint has about 17.3 mm position error and passes a declared 20 mm limit, while the overall calibration remains pending the full held-out set.
Calibration is accepted by independent spatial evidence across the intended workspace, not by its fitted transform or one attractive overlay.
Physical examples
Where this appears in real life
Two taped rulers
Tape one ruler to a cardboard camera and another to a cardboard robot base, then measure the fixed offset and direction between their origins.
A point receives different coordinates on the two rulers, while one rigid rotation and translation should connect every stationary point.
Loose phone bracket
Mark a phone holder's position, nudge the holder slightly, and compare where the same distant wall mark appears before and after.
A small mount rotation causes a larger lateral shift at distance, showing why a moved bracket invalidates a formerly good transform.
Hands-on exercise
Make the idea observable
Use a simulated camera and calibration target or a recorded public calibration dataset with known poses. Keep the transform output disconnected from motion commands.
Declare eye-to-hand or eye-in-hand layout, parent and child frames, transform direction, units, image resolution, intrinsics version, and timestamp source.
Split varied target observations into a fitting set and a held-out validation set before estimating the extrinsic transform.
Estimate the transform and assert rotation orthonormality, determinant near +1, finite translation, expected mount bounds, and consistent units.
Evaluate held-out points or poses across the intended workspace, reporting per-axis error, Euclidean error, sample count, and worst location rather than only an average.
Inject a reversed transform, one target-scale error, and one pose timestamp offset; save the distinct numerical and overlay symptoms for diagnosis.
Version the dataset, solver parameters, transform, validation table, mount state, limitations, and recalibration triggers in one calibration record.
A good result should remain within the declared task tolerance across held-out positions; planted direction, scale, and timing faults should fail different checks instead of becoming one vague calibration score.
The calibration artifact reproduces its transform, passes every declared rigid-transform assertion and held-out gate, and rejects all three planted faults with specific reasons.
Build today
Transform an RGB-D observation into a filtered point cloud and safe grasp or approach target.
Evidence to save
DONE when a comparison table for “Sensor extrinsic calibration” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Using camera-to-base numbers while publishing or applying them as base-to-camera.
Write the parent-child direction explicitly, verify a known point, and invert the full rigid transform only when the consumer truly needs the opposite mapping.
Accepting the transform because reprojection on the fitting images is small.
Reserve varied held-out observations and measure 3D task-space error across the intended workspace and range.
Reusing calibration after the mount, focus, resolution, robot model, or clock relationship changes.
Version those dependencies, define invalidation triggers, check the rigid mount, and rerun the appropriate calibration and validation workflow.
Job connection
How this becomes employable evidence
Commission a fixed or wrist-mounted depth camera, version its frame contract, qualify held-out workspace accuracy, and determine whether a shifted target came from mount movement, transform direction, units, kinematics, or timestamp mismatch.
Relevant target roles
- Robotics Application / ROS 2 Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 10 interview drill
Interview questions: Sensor extrinsic calibration
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Separate intrinsics from extrinsics, compare eye-to-hand with eye-in-hand, and describe the dataset, rigid-transform checks, held-out measurements, and change controls needed before accepting calibration.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What is the difference between intrinsics and extrinsics?
Intrinsics describe pixel-to-ray camera geometry and distortion; extrinsics describe rigid rotation and translation between the camera frame and another frame.
Q2Why use target poses with varied orientation and workspace location?
They make more transform directions observable and test whether the calibration generalizes beyond a narrow, nearly identical set of views.
Q3How can timing error resemble spatial calibration error?
If the camera or robot moves between image acquisition and the pose used for transformation, correct geometry is applied to the wrong physical instant.