Chapter 10 · Turn depth measurements into safe, inspectable 3D targets

Today in the field story

One problem, then the next

Return to the glossy divider, dark mat, and partly hidden tote that started the incident. Run the same target path without filling holes merely to make the display complete. Measure missing-depth regions, pose instability, and disagreement across views; then define when the system should request another viewpoint or reject the target. The mission gains credibility when these awkward scenes remain visible in the report.

Why now: A nominal pipeline says little until known sensor and visibility failures challenge its gate.
Ignore today: Ignore learned inpainting and synthetic perfection; preserve unknown space and bounded recovery choices.
Unlocks next: A bad-case suite that can stop unreliable geometry before planning.

Understand

Build the physical picture first

A hole in a depth image is a report about visibility or signal, so safe perception keeps the hole and asks why instead of painting in a solid answer.

Occlusion happens when a nearer surface blocks the sensor's view of a farther surface. A single depth view therefore measures visible surfaces, not complete objects. Along an object boundary, a matching or averaging window can mix foreground and background and create flying points between them. A partial cloud can move a centroid, shrink an extent, or rotate an oriented box even when every retained point is numerically finite.

Surface material changes the returned signal. Dark material can weaken active light. A mirror-like surface can send it away or return another path. Transparent material can transmit or mix foreground and background. Strong ambient light can reduce projected-pattern contrast, and multiple active sensors can interfere. Range limits and steep viewing angles also matter. These cases may create missing samples, noisy depth, saturated readings, or confidently biased values, so a finite-number check alone is insufficient.

Represent validity before geometry. Check encoding and scale, finite values, sensor range, device confidence where available, left-right or temporal consistency where applicable, and neighborhood discontinuity. Calculate missing-depth rate with a stated region and denominator. Preserve categories such as invalid, out of range, occluded, low confidence, and algorithm rejected when the sensor exposes enough evidence. Do not call all missing pixels obstacles or all of them free space.

A task policy should react to evidence quality. It can reject the target, request another stationary viewpoint, use a different validated sensor, enlarge uncertainty, slow or stop an offline planning decision, or ask an operator. Filling small display holes may help a human see an image, but filled values must remain distinguishable from measured depth and cannot silently pass the metric-geometry gate. Evaluate bad cases by surface, range, angle, occlusion fraction, valid coverage, pose error, and false-accept count.

Words you need

Name each idea precisely

Occlusion: Loss of visibility because a nearer surface blocks light paths to a farther surface.
Physical example:
A cereal box hides the back half of a cup from a front-facing camera.
Flying point: A false depth sample that appears between foreground and background, often near a discontinuity or mixed measurement.
Physical example:
Dots appear suspended between a box edge at 0.8 m and the wall behind it at 2.0 m.
Missing-depth rate: The count of invalid depth samples divided by all samples in a declared evaluation region.
Physical example:
240 invalid pixels among 2,000 target-region pixels gives a 12% missing rate.
Valid coverage: The proportion of a target region supported by depth samples that pass the declared validity checks.
Physical example:
A box detection with valid depth on only 35% of its region may be rejected for pose estimation.
False accept: A bad or unsupported target that incorrectly passes the perception quality gate.
Physical example:
A reflection produces a tidy cluster that the system publishes as a graspable object despite failing the material test.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalDepth, point distance, and rigid alignment

3D perception needs distances and poses that agree across frames.

p = [x,y,z]: a 3D pointUnit: metres (m)
||p||: distance from the frame originUnit: metres (m)
eᵢ: alignment residual for point iUnit: metres (m)

For p = [0.3, 0.4, 0] m, square the components: 0.09 + 0.16 + 0.
The sum is 0.25 m².
Distance is √0.25 = 0.5 m; invalid or missing depth must be excluded before this calculation.

Programmer analogy

Like mapping API objects between schemas, point clouds require an explicit transform; unlike JSON, a wrong transform can still look plausible.

How far is [0, 0, 2] m from the camera origin?

2 m.

The missing-depth rate is

\frac{N_{\mathrm{missing}}}{N_{\mathrm{total}}}=\frac{240}{2000}=12\%.

Back-projection $X=(u-c_x)Z/f_x$ is undefined when depth $Z$ is missing, so the pipeline must mask that sample rather than substitute $Z=0$ .

Use missing evidence to reject an unsupported target

A target region contains 2,000 pixels. The validity checks accept 1,760 and reject 240. The target gate requires at least 92% valid coverage, no more than 3% edge-flying-point candidates, and a finite median depth inside the sensor range.

Calculate missing-depth rate: 240 / 2,000 = 0.12, or 12%.
Calculate valid coverage: 1,760 / 2,000 = 0.88, or 88%.
Compare coverage with the gate: 88% < 92%, so the target already fails even before pose estimation.
Do not back-project the 240 rejected pixels and do not substitute Z = 0; keep their mask and categories.
Record edge-flying-point rate and median valid depth for diagnosis, but do not let those later checks override the failed hard gate.
Return a structured INSUFFICIENT_VALID_DEPTH result with counts, thresholds, frame, stamp, surface condition, and a safe next observation request.

Result

The pipeline rejects the target because only 88% of its region has trusted depth, while retaining enough evidence to compare a changed viewpoint or sensor condition.

What this proves

A clear rejection with denominators is more actionable and safer than a complete-looking point cloud manufactured from missing measurements.

Physical examples

Where this appears in real life

Cup behind a book

Move a book in front of a cup while viewing from one fixed position, then move your head sideways without touching either object.

Look for:

The hidden cup surface is absent from the first view and becomes observable from a second viewpoint; it was unknown, not erased from the world.

Matte, foil, and clear sheet

Compare how a fixed lamp appears on matte card, aluminium foil, and clear plastic while keeping your eyes away from direct glare.

Look for:

Absorption, directional reflection, and transmission create different signal paths, so one sensor model cannot treat the three returns as equally trustworthy.

Hands-on exercise

Make the idea observable

Use a recorded public RGB-D scene or a synthetic renderer with matte, reflective, transparent, near-edge, and partly occluded targets. Produce no robot commands.

Freeze target regions and test conditions, then save raw depth, validity mask, confidence if available, colour image, camera metadata, and environmental notes.
Run the same validity policy on every case and report total pixels, valid pixels, missing rate, edge-invalid rate, depth spread, and target accept or reject.
Create side-by-side overlays for measured valid depth, rejected pixels by category, and any display-only hole fill with a visibly different colour.
Move only one condition at a time—viewpoint, range, angle, ambient light in simulation, or occluder—and compare whether the evidence improves or merely looks smoother.
Plant a dangerous fallback that converts invalid depth to zero, capture the false geometry or metric effect, then remove it and add a regression assertion.
Run the frozen set twice and attach counts, rejection reasons, the largest residual false accept, and the next controlled experiment.

Observe

Different failure surfaces should produce distinct spatial masks or bias patterns, and the target decision should follow declared evidence gates instead of display smoothness.

Done when

The report reproduces all cases, keeps measured and filled data separate, blocks the zero-fill regression, and names the remaining false accept or unsupported surface.

Build today

Transform an RGB-D observation into a filtered point cloud and safe grasp or approach target.

Evidence to save

DONE when a 60–120 second uncut “Occlusion, reflective surfaces, and missing depth” demo links to its command, logs or plots, result count, and honest failure note.

Common mistakes

Catch the wrong mental model

Wrong

Judging depth quality from a smoothly colorized image.

Better

Inspect raw values, validity categories, denominators, spatial masks, known distances, and downstream pose error; display smoothing can hide missing or biased evidence.

Wrong

Filling holes and then forgetting which points were measured.

Better

Keep measured, inferred, and invalid masks separate, and forbid display-only interpolation from entering metric geometry without an independently validated model and policy.

Wrong

Reporting one missing-depth percentage for the whole image.

Better

Measure declared task regions and stratify by material, range, angle, boundary, occlusion, and test condition so failures cannot hide in easy background pixels.

Job connection

How this becomes employable evidence

Build a depth-sensor qualification matrix for warehouse or tabletop materials, then turn recurring invalid, biased, occluded, and interference patterns into runtime quality gates and replay tests.

Relevant target roles

Robotics Deployment, Integration & Validation Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Application / ROS 2 Integration Engineer

Chapter 10 interview drill

Interview questions: Occlusion, reflective surfaces, and missing depth

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Explain why missing depth is not zero, how occlusion and reflective or transparent surfaces alter geometry, and which denominators and false-accept tests you would use before permitting a 3D target.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why can a partial but finite point cloud still give a wrong object pose?

Model interview answer

Occlusion changes which surfaces contribute, so the centroid, extent, normals, and oriented axes can shift even when all remaining coordinates are finite.

Q2What denominator belongs in missing-depth rate?

Model interview answer

All pixels in the explicitly declared evaluation region under the same test condition, with invalid pixels as the numerator.

Q3When is hole filling acceptable in this chapter's pipeline?

Model interview answer

It may be used for a clearly labelled display, but filled values remain separate and cannot silently satisfy the measured-depth gate for metric target geometry.