Chapter 10
Turn depth measurements into safe, inspectable 3D targets
Move from flat image regions to measured geometry: understand how depth sensors fail, back-project valid pixels into a framed point cloud, reduce and segment that cloud, estimate an object pose and approach frame, calibrate the sensor to the robot, and reject targets when the evidence is too weak for a safe downstream action.
Before you start
- Use Week 9 image coordinates, camera intrinsics, distortion correction, timestamps, confidence thresholds, overlays, and frozen evaluation examples.
- Follow a tf2 frame tree and compose a rigid transform while keeping metres, pixels, axes, direction, and acquisition time explicit.
- Run small Python examples and inspect arrays, images, point clouds, logs, and saved result tables.
- Keep all exercises offline, recorded, or simulated; no perception result in this chapter authorizes motion on a powered robot.
By the end
- Compare passive stereo, structured-light, and time-of-flight depth by their measured signal, useful conditions, and characteristic invalid or biased returns.
- Back-project a valid depth pixel with camera intrinsics, publish or inspect its ROS camera metadata, and preserve frame, timestamp, encoding, scale, and validity.
- Build a point cloud, apply a task-sized voxel filter, transform it at the measurement time, and explain which geometric detail the filter removes.
- Extract a tabletop with a bounded plane model, refine overlapping cloud alignment with ICP, and reject registrations that lack a credible initial guess or geometric support.
- Separate axis-aligned extent, oriented extent, 6D object pose, symmetry, and the task-specific approach frame consumed by a planner.
- Estimate and validate a camera-to-robot extrinsic transform on held-out observations rather than accepting a visually plausible overlay.
- Measure missing-depth patterns and run an end-to-end target pipeline whose quality gates, overlays, rejection reasons, timing, and bad cases are reproducible.
The field story
Recover the blue tote from the glass-box shelf
A warehouse robot has one modest assignment: identify a blue parts tote on a waist-high shelf and present a trustworthy approach pose to the planner. The RGB image looks convincing, yet the shelf contains a glossy divider, a dark rubber mat, and a second tote partly hidden behind a carton. Those surfaces turn a tidy rectangle into an engineering argument. This week follows one recorded shelf scene from raw depth through framed geometry, rejecting any target whose measurement cannot support the next decision.
You are the perception engineer on the morning after a near miss. The previous build drew a perfect green box around the tote, but its depth came from the divider behind it. Your job is not to make the overlay prettier. You must account for invalid returns, transform points at their acquisition time, isolate the support plane, estimate a pose that respects symmetry, validate the camera extrinsic, and preserve bad cases. The mission succeeds only when another engineer can reproduce both acceptance and rejection from saved evidence.
- Why this chapter now
Week 9 established image measurements and camera intrinsics; manipulation now needs metric geometry, framed poses, and evidence that a visually plausible target is physically usable.
- Ignore for now
Do not optimize a neural depth model, tune a powered grasp, or chase dense reconstruction quality. Work with recorded or simulated data and one bounded target contract.
- This unlocks
A trustworthy 3D target becomes the input to MoveIt planning, grasp approach design, calibration checks, and later dataset and policy work.
- Proof you will leave with
Submit raw and filtered fixtures, frame and timestamp metadata, calibration revision, local-normal stability, overlays, numeric residuals, rejection counts, one nominal case, at least two named bad cases, and the exact rerun command; label fixed-camera extrinsic evidence separately from any unperformed hand–eye calibration.
Environment contractRepository-supported Node.js 22.13.0 or newer runs the dependency-free starter. The learner-created full lab targets ROS 2 Jazzy with Gazebo Harmonic plus recorded or simulated camera data; this repository does not bundle the camera fixture or simulator workspace, and no perception result authorizes powered motion.
- Compatibility boundary
Keep depth encoding, scale, CameraInfo, tf2 direction, point-cloud schema, and calibration identity explicit. Camera drivers and point-cloud tooling can differ, so validate the actual message contract rather than assuming package parity.
- Smoke check
Run
node week-10-depth-quality-gate.mjs, then replay one saved depth fixture and confirm the point count, frame, timestamp, accepted target, and rejection reason are identical on a second run.- Contract reviewed
2026-07-25
- Runtime evidence
The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
- Drift risk
medium
Today in the field story
One problem, then the next
The blue tote is visible, but the glossy divider returns a confident distance from the wrong surface while the dark mat produces holes. Begin the mission by classifying what stereo, structured light, or time of flight actually measures. Build a validity mask before calculating geometry, and record why each missing or biased pixel is unknown rather than silently turning it into a nearby obstacle or a zero-distance target.
- Why now
A 3D pipeline is only as credible as the sensor evidence entering it.
- Ignore today
Ignore dense scene reconstruction and sensor shopping; compare failure mechanisms on the saved shelf scene.
- Unlocks next
A bounded valid-depth set that can be converted into framed points.
Understand
Build the physical picture first
A depth camera is a ruler that asks light a question; different rulers ask differently, and a blank answer is not the same as zero distance.
A normal color image tells which light reached each pixel but does not directly tell how far the surface was. A depth image adds one range value for each usable pixel. Passive stereo uses two separated cameras and matches the same feature in both images. Structured light projects a known pattern and measures how the pattern shifts. Time-of-flight sensors estimate travel time or phase of emitted light. All three produce estimates, not guaranteed truth.
For a rectified stereo pair, corresponding points lie on the same image row. Their horizontal separation is disparity d. With focal length f in pixels and camera baseline B in metres, depth is Z = fB / d. Large disparity means a nearby point; small disparity means a distant point. The equation assumes correct calibration, rectification, correspondence, units, and nonzero disparity. A neat number from broken assumptions remains broken.
Once a pixel (u, v) has valid depth Z, the pinhole model back-projects it into the optical camera frame: X = (u - cx)Z / fx and Y = (v - cy)Z / fy. In the standard ROS optical-frame convention, x points right in the image, y points down, and z points forward. The depth image and its CameraInfo must describe the same camera geometry, resolution, frame, and acquisition time or the reconstructed point will be displaced.
Each sensing method has recognizable weak cases. Stereo struggles on plain or repeating texture, narrow edges, occlusion boundaries, and poor left-right matches. Active projected light can be washed out by strong ambient light or disturbed by another projector. Dark, shiny, transparent, distant, or steeply angled surfaces can absorb, redirect, mix, or weaken returns. Keep a validity mask and a reason for rejection; never turn invalid depth into a confident obstacle or target.
Words you need
Name each idea precisely
- Depth image
A pixel grid whose valid entries estimate distance along the camera viewing direction under a declared encoding and scale.
Physical example:A cup pixel may store 820 in a millimetre image, meaning about 0.82 m only if the driver contract says the unit is millimetres.
- Disparity
The horizontal pixel difference between matching points in a rectified stereo pair.
Physical example:A nearby finger shifts much farther between left-eye and right-eye views than a far wall does.
- Baseline
The measured separation between the optical centres of the two cameras in a stereo pair.
Physical example:Two cameras mounted 0.10 m apart have a 0.10 m baseline when that value uses metres consistently.
- Back-projection
The conversion of a pixel plus valid depth and intrinsics into a 3D point in the camera frame.
Physical example:A pixel 100 columns right of the principal point becomes a positive camera-x coordinate.
- Validity mask
A same-sized map stating which depth entries may be used after range, encoding, confidence, and finite-value checks.
Physical example:Pixels on a glass rim are marked invalid rather than filled with zero and treated as real geometry.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalDepth, point distance, and rigid alignment
3D perception needs distances and poses that agree across frames.
- p = [x,y,z]
- a 3D pointUnit: metres (m)
- ||p||
- distance from the frame originUnit: metres (m)
- eᵢ
- alignment residual for point iUnit: metres (m)
For p = [0.3, 0.4, 0] m, square the components: 0.09 + 0.16 + 0.
The sum is 0.25 m².
Distance is √0.25 = 0.5 m; invalid or missing depth must be excluded before this calculation.
Like mapping API objects between schemas, point clouds require an explicit transform; unlike JSON, a wrong transform can still look plausible.
How far is [0, 0, 2] m from the camera origin?
2 m.
Stereo depth is
Back-projecting pixel with and gives . Neither equation models reflective-surface failure.
Recover one stereo point and state its limits
A rectified stereo pair has f = fx = fy = 500 px, baseline B = 0.10 m, and principal point (cx, cy) = (320, 240) px. A match appears at (u, v) = (420, 265) px with disparity d = 25 px.
Check the contract first: focal length and disparity are in pixels, baseline is in metres, the pair is rectified, and disparity is positive and nonzero.
Calculate depth:
Z = fB / d = 500 × 0.10 / 25 = 2.00 m.Calculate camera x:
X = (420 - 320) × 2.00 / 500 = 0.40 m.Calculate camera y:
Y = (265 - 240) × 2.00 / 500 = 0.10 m.Report the point as
(0.40, 0.10, 2.00) min the optical camera frame at the image acquisition time.Mark the result invalid if correspondence, rectification, calibration, depth range, or left-right consistency fails; the arithmetic cannot certify those inputs.
The valid teaching case back-projects to a point 0.40 m right, 0.10 m down, and 2.00 m forward in the optical camera frame.
Depth geometry is a chain of measured assumptions, so the output needs a frame, time, unit, and validity decision as much as it needs three coordinates.
Physical examples
Where this appears in real life
Two-eye finger test
Hold one finger near your face and alternately close each eye, then repeat with the finger farther away while keeping the background fixed.
The near finger appears to jump farther relative to the background, showing that stereo disparity grows as depth decreases.
Pattern on paper and foil
Shine a patterned light onto matte paper, crumpled foil, and a clear plastic sheet without using the result to control anything.
The matte surface preserves a readable pattern while reflection and transmission can redirect or erase parts of it, which explains invalid active-depth regions.
Hands-on exercise
Make the idea observable
Use OpenCV's saved stereo sample or another public, non-sensitive rectified pair. Work offline and keep any robot command path disabled.
Record the image size, focal length, principal point, baseline, disparity scale, valid disparity range, and the exact input files before computing depth.
Create a disparity image and retain the raw disparity plus a display copy; do not use the normalized display colours as metric depth.
Choose three pixels in textured, textureless, and occlusion-edge regions, run the validity checks, and back-project only those that pass.
Change one matcher setting at a time, such as block size or uniqueness ratio, and compare valid count, speckles, edge bleeding, and one known-distance error.
Plant one wrong baseline unit or unrectified input, capture the plausible-looking but incorrect output, then restore the recorded contract.
Save a small table with pixel, disparity, calculated depth, validity reason, frame, timestamp source, and expected physical limitation.
Reliable regions should have coherent disparity and plausible geometry; weak texture and boundaries should expose holes or unstable matches rather than silently becoming exact surfaces.
The saved example reproduces one correct back-projection, one rejected sample, and one documented unit or rectification failure from the same declared inputs.
Build today
Transform an RGB-D observation into a filtered point cloud and safe grasp or approach target.
Evidence to save
DONE when the learning log explains “Depth cameras, stereo, structured light, and failure modes” in five precise points and a checked example produces the predicted output.
Common mistakes
Catch the wrong mental model
Reading a colourized depth display as if its colour values were physical metres.
Use the raw depth or disparity encoding and its documented scale; keep visualization normalization out of the metric computation.
Replacing every invalid, NaN, saturated, or unmatched depth value with zero.
Preserve a validity mask and rejection reason because missing depth is unknown evidence, while zero may be interpreted as a real point at the sensor.
Applying Z = fB/d to an unrectified pair or mixing millimetres and metres.
Verify calibration and rectification, require positive usable disparity, and normalize the baseline and output to declared physical units.
Job connection
How this becomes employable evidence
Bring up a stereo or RGB-D stream, verify calibration, encodings, optical frames, timestamps, depth scale, and invalid-pixel behavior before allowing its measurements into collision checking or target estimation.
Relevant target roles
- Robotics Application / ROS 2 Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 10 interview drill
Interview questions: Depth cameras, stereo, structured light, and failure modes
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Compare stereo, structured light, and time of flight; derive Z = fB/d; then explain which metadata and failure checks you require before back-projecting a ROS depth image.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What happens to stereo depth when disparity becomes smaller but `f` and `B` stay fixed?
Calculated depth grows because Z = fB/d; small disparity also makes a one-pixel matching error more damaging to the depth estimate.
Q2Which frame contains a point immediately after pinhole back-projection?
The camera optical frame described by the image and CameraInfo contract, not automatically base_link, map, or a grasp frame.
Q3Why is a missing depth sample not a measured distance of zero?
It means the sensor pipeline could not produce a trusted range under its current conditions; zero is a numerical value that downstream code may treat as real geometry.
Chapter references
- OpenCV — Depth Map from Stereo ImagesThe official disparity relationship, stereo block-matching example, and the texture, speckle, disparity-range, and uniqueness controls that reveal why stereo depth can be invalid.
- OpenCV — Camera Calibration and 3D ReconstructionThe pinhole projection model, intrinsic and extrinsic parameters, distortion, stereo calibration and rectification, triangulation, reprojection, and pose functions used in the chapter.
- Open3D — Point cloud tutorialOfficial behavior for voxel downsampling, normal estimation, cropping, bounding volumes, clustering, and RANSAC plane segmentation on point clouds.
- Open3D — ICP registration tutorialThe correspondence-and-update loop, required rough initial alignment, point-to-point and point-to-plane objectives, fitness, inlier RMSE, and convergence limits.
- ROS 2 Jazzy common_interfaces — sensor_msgs definitionsPrimary CameraInfo, Image, PointCloud2, and PointField contracts for acquisition time, optical frame, calibration matrices, encodings, organized cloud shape, fields, and invalid samples.