Chapter 10 · Turn depth measurements into safe, inspectable 3D targets

Today in the field story

One problem, then the next

The robot-frame cloud now contains shelf, divider, carton, and tote in one heap. Fit the support plane within a declared tolerance, remove it without swallowing the tote base, and cluster what remains. Use ICP only to refine a plausible alignment, not to rescue an arbitrary initial guess. The recurring question is whether the blue-tote cluster survives every transformation with enough support to justify a target.

Why now: Object geometry becomes useful only after dominant structure and registration assumptions are tested.
Ignore today: Ignore general-purpose scene understanding; solve the bounded tabletop extraction and one alignment check.
Unlocks next: An isolated target cluster with residuals and explicit rejection thresholds.

Understand

Build the physical picture first

Table extraction finds the broad supporting sheet, while ICP slides one already-near 3D tracing onto another; neither method knows which physical object you intended.

A tabletop scene often contains one large plane plus objects above it. A plane model ax + by + cz + d = 0 describes points lying on a flat surface. RANSAC repeatedly samples a small set, proposes a plane, and counts points within a distance threshold as inliers. Open3D's segment_plane exposes the distance threshold, sample count, and iteration count. The largest plane is only a mathematical winner; it might be a wall or floor unless orientation, height, region, and support rules identify the table.

Tabletop extraction usually crops to a known workspace, removes invalid and distant points, estimates the support plane, checks its normal and height, then keeps candidate points on the allowed side and within an object-height band. A generous plane threshold can absorb a thin object. A tight threshold can leave noisy table patches as false objects. Record the inlier ratio and rejected geometry while tuning against labelled scenes, not one clean view.

Registration estimates the rigid transform that aligns overlapping clouds. Iterative Closest Point starts from a rough initial transform, finds nearby correspondences under a distance limit, updates the transform to reduce an error, and repeats. Point-to-point ICP reduces coordinate distances. Point-to-plane ICP uses target surface normals and can converge faster on suitable surfaces. ICP is a local refinement method: poor initial alignment, weak overlap, repeated geometry, bad normals, moving objects, or an oversized correspondence distance can produce a convincing wrong alignment.

Open3D reports alignment evidence such as a transformation, fitness, inlier RMSE, and correspondence set. A lower inlier RMSE does not prove the transform is correct if only a tiny or ambiguous part overlaps. Gate registration with expected motion bounds, overlap or fitness, residual, correspondence count, determinant and rigidity checks, and an external geometric test. Keep the initial and final overlays so a reviewer can see what the scalar metrics omit.

Words you need

Name each idea precisely

Plane model: Four coefficients defining a flat set of 3D points through ax + by + cz + d = 0.
Physical example:
An ideal horizontal tabletop at z = 0.75 m can be represented by z - 0.75 = 0 in its chosen frame.
RANSAC: A repeated sample-fit-count method that searches for a model supported by many inliers despite outliers.
Physical example:
Many tabletop points vote for one plane while points on cups and boxes remain outside its distance band.
Plane inlier: A point whose perpendicular distance to the proposed plane is no more than the declared threshold.
Physical example:
A point 0.012 m above an estimated plane is accepted when the threshold is 0.020 m.
ICP: Iterative Closest Point, a local registration process that alternates correspondence search and transform refinement.
Physical example:
A second scan of a stationary box starts a few centimetres offset and is refined onto the first scan.
Inlier RMSE: The root-mean-square residual over correspondences accepted by the current registration threshold.
Physical example:
An RMSE of 0.008 m summarizes accepted residuals but says nothing about surfaces that never corresponded.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalDepth, point distance, and rigid alignment

3D perception needs distances and poses that agree across frames.

p = [x,y,z]: a 3D pointUnit: metres (m)
||p||: distance from the frame originUnit: metres (m)
eᵢ: alignment residual for point iUnit: metres (m)

For p = [0.3, 0.4, 0] m, square the components: 0.09 + 0.16 + 0.
The sum is 0.25 m².
Distance is √0.25 = 0.5 m; invalid or missing depth must be excluded before this calculation.

Programmer analogy

Like mapping API objects between schemas, point clouds require an explicit transform; unlike JSON, a wrong transform can still look plausible.

How far is [0, 0, 2] m from the camera origin?

2 m.

Point-to-plane distance here is

d=\lvert z_{\text{point}}-z_{\text{plane}}\rvert=\lvert0.762-0.750\rvert=0.012\ \mathrm{m}.

Since $d<0.015\ \mathrm{m}$ , the point is an inlier under that threshold; ICP RMSE changes from $0.040$ to $0.009\ \mathrm{m}$ .

Classify plane points, then judge an ICP result

An estimated tabletop is z = 0.75 m with distance threshold 0.02 m. Three test points have z values 0.742, 0.762, and 0.810 m. A later ICP run reports fitness 0.68, inlier RMSE 0.009 m, and translation 0.032 m; the gate requires fitness at least 0.60, RMSE at most 0.012 m, and translation at most 0.05 m.

Compute absolute plane distance for the first point: |0.742 - 0.75| = 0.008 m, so it is a plane inlier.
For the second point, |0.762 - 0.75| = 0.012 m, so it is also an inlier under the 0.02 m threshold.
For the third point, |0.810 - 0.75| = 0.060 m, so it remains outside the tabletop plane and may belong to an object.
Check ICP fitness: 0.68 ≥ 0.60, so the overlap gate passes for this frozen case.
Check residual and motion: 0.009 ≤ 0.012 m and 0.032 ≤ 0.05 m, so both declared numeric gates pass.
Accept only as a registration candidate, then inspect the initial/final overlay and a held-out feature because these three metrics cannot rule out a symmetric wrong alignment.

Result

Two samples join the support plane, one remains a raised candidate, and the ICP result clears the numeric gates but still requires geometric validation.

What this proves

Thresholds classify evidence under a declared model; passing them narrows risk but does not give the algorithm knowledge of object identity or task intent.

Physical examples

Where this appears in real life

Tray under scattered blocks

Place several small blocks on a flat tray and imagine dots sampled from every visible surface.

Look for:

Most tray dots support one broad plane, while the raised block dots should remain outside a carefully chosen plane-distance band.

Align two transparent tracings

Trace part of the same textured object on two transparent sheets, start them slightly offset, and repeatedly match nearby marks before nudging one sheet.

Look for:

A close start and distinctive overlap lead to the intended alignment; a far start or repeated pattern can settle on the wrong matching region.

Hands-on exercise

Make the idea observable

Generate or load a fixed tabletop point cloud with one thin object, one taller object, and a second partially overlapping scan. Record the random seed and do not command hardware.

Crop the declared workspace, report finite points and bounds, and color the raw support surface and raised objects for reference.
Run plane segmentation across three distance thresholds; save plane coefficients, normal direction, inlier count, and which labelled object points were wrongly absorbed.
Add orientation and height checks so a larger wall-like plane cannot silently replace the intended support plane.
Register the second scan from a good initial transform and save fitness, inlier RMSE, correspondence count, final transform, runtime, and overlay.
Repeat from a deliberately poor initial transform or with reduced overlap, require the quality gate to reject the result, and capture the misleading metric or visual symptom.
Restore the valid inputs and assert expected table height, minimum object points, rigid rotation, transform bounds, and deterministic pass or rejection status.

Observe

Plane thresholds should show a visible tradeoff between noisy table leftovers and erased thin objects; ICP should refine the close case and explicitly reject the unsupported case.

Done when

The same command reproduces one accepted tabletop and registration result plus one causally explained failure, with expected and observed metrics attached.

Build today

Transform an RGB-D observation into a filtered point cloud and safe grasp or approach target.

Evidence to save

DONE when a deterministic “Plane segmentation, ICP registration, and tabletop extraction” failure test reports expected versus actual behavior and passes after the documented fix.

Common mistakes

Catch the wrong mental model

Wrong

Calling the largest RANSAC plane the tabletop without checking its orientation, height, or workspace.

Better

Gate the plane against the expected support region, normal direction, height range, inlier support, and downstream object evidence.

Wrong

Increasing the plane distance threshold until the table looks clean.

Better

Evaluate labelled thin objects and noisy table patches together because a wide threshold can erase the geometry the robot needs.

Wrong

Starting ICP from identity for every scene and trusting the smallest RMSE.

Better

Supply a credible initial transform, require adequate overlap and bounded motion, and validate the final transform with overlays and held-out geometry.

Job connection

How this becomes employable evidence

Diagnose a tabletop picking scene where a low part disappears or two scans misalign, separating crop, plane threshold, normal, initial transform, overlap, correspondence, and frame faults with replayable evidence.

Relevant target roles

Robotics Software Engineer — ROS 2 / AMR
Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer

Chapter 10 interview drill

Interview questions: Plane segmentation, ICP registration, and tabletop extraction

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Explain how RANSAC plane segmentation and ICP solve different problems, why each needs task-specific gates, and how a low registration residual can still accompany a wrong physical alignment.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why can the largest detected plane be the wrong support surface?

Model interview answer

RANSAC maximizes geometric support, not task meaning; a wall or floor may contain more inliers than the tabletop.

Q2What two operations repeat inside basic ICP?

Model interview answer

It finds correspondences under the current transform, then updates the transform to reduce the chosen correspondence error.

Q3Why must fitness and inlier RMSE be read together with an overlay and motion bounds?

Model interview answer

Residual covers accepted correspondences and fitness summarizes overlap, but partial or repeated geometry can still support an incorrect rigid transform.