Chapter 10 · Turn depth measurements into safe, inspectable 3D targets

Today in the field story

One problem, then the next

Yesterday’s mask leaves a sparse but defensible patch on the tote. Today each accepted pixel becomes a metric point using the saved intrinsics, then enters the robot frame through the transform valid at capture time. A voxel filter may make the cloud manageable, but its cell size can erase the tote lip. Preserve raw counts and show exactly which geometric detail disappears before the filtered cloud proceeds.

Why now: Planning cannot consume pixels; it needs metric points with frame and time ownership.
Ignore today: Ignore map-scale fusion and global registration; transform one synchronized shelf observation.
Unlocks next: A compact point cloud whose dimensions and coordinate meaning can be inspected.

Understand

Build the physical picture first

A point cloud is a bag of tiny framed measurements, and a voxel filter replaces nearby grains with one representative grain without repairing where the bag was placed.

Back-projecting many valid depth pixels creates a point cloud. Each sample usually has x, y, and z and may also carry colour, intensity, confidence, or another field. A ROS PointCloud2 stores a header, height, width, named fields, byte layout, and a density flag. An organized cloud can preserve the image row-column arrangement; an unorganized cloud is commonly one row. Always inspect the actual fields and layout instead of assuming every point has the same extras.

Point clouds become large quickly, so voxel downsampling divides 3D space into cubes and emits one representative point for each occupied cube. Open3D's basic voxel operation averages the points inside each occupied voxel. With voxel size s, a teaching index along x is floor(X/s). The size must come from the smallest task-relevant geometry and noise level. A large voxel speeds later work but can erase a thin handle, merge two close objects, or move an edge.

Every cloud belongs to the frame and time in its header. To use camera points near a robot, transform them through the calibrated frame tree at the acquisition timestamp. A current transform applied to an old image can misplace a point when the camera or robot moved between capture and processing. Downsampling before or after a rigid transform gives similar positions in an ideal case, but cropping, voxel grid origin, floating-point details, and changing transforms can alter membership, so record the order.

A useful cloud stage publishes or saves more than pretty dots. Track input count, invalid count, cropped count, output count, voxel size, bounds, frame, stamp, transform age, and processing time. Then test a known point numerically. A cloud can look like the right object while being mirrored, scaled by 1,000, expressed in the wrong frame, or delayed enough to be unsafe.

Words you need

Name each idea precisely

Point cloud: A collection of sampled 3D points whose coordinates are meaningful only with a declared frame, unit, and acquisition time.
Physical example:
Thousands of valid tabletop depth pixels become thousands of camera-frame surface samples.
PointCloud2: The ROS sensor message that describes a binary array of points using dimensions, named fields, byte strides, a header, and validity metadata.
Physical example:
A cloud can contain x, y, z, and packed colour fields while its header says the samples were acquired in camera_depth_optical_frame.
Voxel: A small cube in a regular 3D grid used to group nearby points.
Physical example:
Several noisy samples inside one 1 cm cube become one averaged representative sample.
Rigid transform: A rotation and translation that change a point's coordinate frame without changing the object's physical size or shape.
Physical example:
The same cup point receives different coordinates in camera and base frames but remains the same physical point.
Organized cloud: A point cloud whose height and width retain a grid relationship such as the source depth image layout.
Physical example:
Point row 100, column 220 corresponds to depth-image pixel row 100, column 220 when the producer preserves organization.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalDepth, point distance, and rigid alignment

3D perception needs distances and poses that agree across frames.

p = [x,y,z]: a 3D pointUnit: metres (m)
||p||: distance from the frame originUnit: metres (m)
eᵢ: alignment residual for point iUnit: metres (m)

For p = [0.3, 0.4, 0] m, square the components: 0.09 + 0.16 + 0.
The sum is 0.25 m².
Distance is √0.25 = 0.5 m; invalid or missing depth must be excluded before this calculation.

Programmer analogy

Like mapping API objects between schemas, point clouds require an explicit transform; unlike JSON, a wrong transform can still look plausible.

How far is [0, 0, 2] m from the camera origin?

2 m.

For coordinate $X=0.42\,\mathrm{m}$ and voxel size $s=0.10\,\mathrm{m}$ ,

i=\left\lfloor\frac{X}{s}\right\rfloor=\lfloor4.2\rfloor=4.

A back-projected coordinate example is $(420-320)\times2/500=0.4\,\mathrm{m}$ . Voxelization reduces samples but cannot fix a wrong frame.

Downsample and transform a known camera point

A camera-frame point is (0.42, 0.11, 1.20) m. The voxel size is 0.10 m. For a deliberately aligned-axis check, the camera origin is translated by (0.20, -0.05, 0.30) m in base_link with no rotation.

Compute the teaching voxel indexes: floor(0.42/0.10)=4, floor(0.11/0.10)=1, and floor(1.20/0.10)=12.
Label the voxel (4, 1, 12) and remember that Open3D emits the average of all points in that occupied voxel, not necessarily its centre.
Check the transform direction: the stated translation maps camera-frame coordinates into base_link, not the reverse.
Because the axes are aligned in this small example, add x: 0.42 + 0.20 = 0.62 m.
Add y and z: 0.11 - 0.05 = 0.06 m, and 1.20 + 0.30 = 1.50 m.
Report (0.62, 0.06, 1.50) m in base_link at the measurement time, and use a full rotation-plus-translation transform in the real pipeline.

Result

The known sample belongs to teaching voxel (4, 1, 12) and maps to (0.62, 0.06, 1.50) m in the aligned base-frame example.

What this proves

Voxelization changes sampling density, while a transform changes coordinates; neither operation validates the other's parameters.

Physical examples

Where this appears in real life

Beans in an ice-cube tray

Scatter several beans into a tray and replace all beans in each occupied compartment with one bean placed near their average position.

Look for:

The number of samples drops while the broad occupied shape remains; a small feature disappears when its compartment is shared with a larger surface.

Same sticker, two rulers

Measure a sticker from a camera-shaped paper origin and from a robot-base paper origin whose axes and origin are drawn differently.

Look for:

The coordinates change with the ruler, but the sticker does not move; the labelled transform connects the two descriptions.

Hands-on exercise

Make the idea observable

Use a saved public depth image or generate a synthetic plane plus two boxes. Use Open3D for cloud operations and a declared 4×4 transform; keep hardware disconnected.

Back-project valid samples and print point count, finite-value count, bounds, units, frame name, and timestamp before viewing the cloud.
Save one known point and independently calculate its expected transformed coordinate as a numerical assertion.
Downsample copies at two voxel sizes, then record output count, processing time, box dimensions, and whether the smaller object remains separable.
Apply the declared transform at the recorded acquisition time and draw both camera and base axes beside the transformed cloud.
Inject a 1,000× scale error and a reversed transform, capture their bounds and axis symptoms, then restore the correct contract.
Export the exact input, transform, parameters, metrics, and one before-after view so another person can reproduce the boundary case.

Observe

A sensible voxel size reduces repeated surface samples without deleting task-critical geometry, and the asserted known point lands at the expected base-frame coordinate.

Done when

One command reproduces the original and two downsampled clouds, passes the known-point transform assertion, and exposes both planted frame or scale failures.

Build today

Transform an RGB-D observation into a filtered point cloud and safe grasp or approach target.

Evidence to save

DONE when “Point clouds, voxel filters, and coordinate transforms” runs from one documented command and the nominal plus boundary outputs are attached.

Common mistakes

Catch the wrong mental model

Wrong

Treating a larger voxel size as a free performance improvement.

Better

Compare the voxel size with the smallest required object feature and report both speed and geometric loss before choosing it.

Wrong

Changing the cloud's frame label without transforming its coordinates.

Better

Apply the correct rigid transform at the measurement timestamp, verify a known point, and only then publish the destination frame.

Wrong

Assuming the binary PointCloud2 layout always contains contiguous float x, y, z, and RGB values.

Better

Read the message's fields, offsets, datatypes, point step, row step, dimensions, and density contract with supported iterators or libraries.

Job connection

How this becomes employable evidence

Build a ROS 2 point-cloud stage that preserves sensor metadata, meets a latency budget, chooses a task-sized voxel resolution, and rejects stale, malformed, mis-scaled, or untransformable input.

Relevant target roles

Robotics Software Engineer — ROS 2 / AMR
Robotics Application / ROS 2 Integration Engineer
Robotics Deployment, Integration & Validation Engineer

Chapter 10 interview drill

Interview questions: Point clouds, voxel filters, and coordinate transforms

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Describe PointCloud2 as a data contract, explain voxel downsampling and its information loss, and show how you would prove a cloud was transformed with the correct direction and timestamp.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What does Open3D's basic voxel downsampling emit for one occupied voxel?

Model interview answer

One representative point formed by averaging the input points that fall inside that voxel.

Q2Can voxel filtering correct a cloud expressed in the wrong coordinate frame?

Model interview answer

No. It changes sampling density; the frame error requires the correct calibrated, time-valid rigid transform.

Q3Why keep point count and bounds before opening a 3D viewer?

Model interview answer

They quickly expose empty data, invalid values, unit errors, extreme outliers, and implausible scale that a pleasing view may hide.