Chapter 10 · Turn depth measurements into safe, inspectable 3D targets
Today in the field story
One problem, then the next
A clean cluster still does not tell the arm how to arrive. Compare an axis-aligned box, an oriented box, a six-degree pose, and the approach frame required above the tote opening. Estimate local surface normals only where neighborhood support is stable, orient them consistently, and reject edge or missing-depth normals. The tote’s symmetry can still flip an axis, so preserve ambiguous alternatives instead of presenting one unstable pose as truth.
- Why now
The next consumer needs a physically meaningful approach, not merely a compact bounding volume.
- Ignore today
Ignore grasp-force tuning and arm execution; define pose semantics and ambiguity offline.
- Unlocks next
A candidate approach frame that MoveIt can later validate against the whole robot.
Understand
Build the physical picture first
A box describes occupied extent, a pose describes an object's frame, and an approach frame describes how a tool should arrive; they answer three different questions.
After removing the support plane, nearby points can be grouped into object candidates. An axis-aligned bounding box uses the current frame's x, y, and z axes, so its size changes when a long object rotates. An oriented bounding box rotates to follow the point distribution and can be tighter, but noise, partial views, and symmetric shapes can flip its axes. Box extent is useful for clearance and rough size; it is not automatically a stable object pose.
A 6D pose combines 3D position with 3D orientation. The name describes six degrees of freedom, not a requirement to store exactly six numbers; ROS commonly represents orientation with a quaternion. Pose must name the parent and child frames and the transform direction. A cylinder may have a well-measured centre while rotation about its own axis is unobservable. A rectangular object seen from one side can also admit a flipped or mirrored hypothesis.
A grasp or approach frame is task-specific. Its origin may be a chosen grasp point rather than the box centre. One axis can point along the tool's approach direction, another along the gripper closing direction, and the third completes a right-handed frame. The convention must match the motion-planning and tool configuration. Before planning, check reachability, pre-grasp offset, finger clearance, support-plane clearance, collision geometry, symmetry alternatives, and pose quality.
Perception should publish a candidate, not an unquestionable command. Include object and frame identifiers, source stamp, dimensions, pose, uncertainty or quality measures, symmetry class, valid-until policy, and rejection reason. If the input changes or cancellation arrives, stop generating new candidates and invalidate queued ones. A motion system still needs its own collision, limits, planning, execution, and safety checks.
Words you need
Name each idea precisely
- Axis-aligned bounding box
A box whose edges remain parallel to the axes of the frame containing the cloud.
Physical example:A diagonal pencil needs a much wider base-frame AABB than the same pencil placed parallel to x.
- Oriented bounding box
A box whose axes are estimated to follow the point set rather than staying fixed to the parent frame.
Physical example:A rotated shoebox can receive a tighter box that follows its long side.
- 6D pose
A rigid object's estimated three-dimensional position and orientation relative to a named parent frame.
Physical example:A part can be 0.6 m ahead of the robot while rotated 30 degrees around a declared axis.
- Approach frame
A task-defined coordinate frame that tells a tool where to arrive and how its approach and closing axes should point.
Physical example:A gripper first moves to a pre-grasp point 0.10 m opposite the approach axis, then follows that axis toward the part.
- Pose symmetry
A condition where multiple orientations produce the same observable object shape or the same valid task result.
Physical example:Rotating an unmarked cylinder around its long axis may not change what the camera can distinguish.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalDepth, point distance, and rigid alignment
3D perception needs distances and poses that agree across frames.
- p = [x,y,z]
- a 3D pointUnit: metres (m)
- ||p||
- distance from the frame originUnit: metres (m)
- eᵢ
- alignment residual for point iUnit: metres (m)
For p = [0.3, 0.4, 0] m, square the components: 0.09 + 0.16 + 0.
The sum is 0.25 m².
Distance is √0.25 = 0.5 m; invalid or missing depth must be excluded before this calculation.
Like mapping API objects between schemas, point clouds require an explicit transform; unlike JSON, a wrong transform can still look plausible.
How far is [0, 0, 2] m from the camera origin?
2 m.
The box top and pre-grasp heights are
Turn object extent into a checked pre-grasp candidate
A segmented box has oriented dimensions 0.20 × 0.10 × 0.05 m, centre (0.60, 0.10, 0.78) m in base_link, and its top is 0.05 m above the support. The declared approach axis points downward, with a 0.10 m pre-grasp offset upward.
Compute box volume:
0.20 × 0.10 × 0.05 = 0.001 m³; treat it as an extent summary, not object mass or certainty.Keep the candidate origin at
(0.60, 0.10, 0.78) mand attach the measured orientation with its frame direction and timestamp.Define the tool approach axis as downward in
base_link; confirm the other two axes complete the planner's right-handed tool convention.Move opposite the downward approach by 0.10 m, giving pre-grasp position
(0.60, 0.10, 0.88) mfor this aligned teaching case.Check that the 0.10 m object width fits inside the open gripper with required side clearance and that the support plane remains outside the planned finger volume.
Publish the target as a candidate only if pose age, symmetry handling, clearance, workspace, and quality gates pass; otherwise publish a structured rejection.
The geometry yields a pre-grasp candidate 0.10 m above the object, but acceptance still depends on orientation convention, clearance, freshness, and planning checks.
A bounding box can help propose a task frame, yet the final approach is an explicit engineering decision with safety and uncertainty gates.
Physical examples
Where this appears in real life
Diagonal shoebox on graph paper
Place a shoebox diagonally on graph paper, outline a box that follows the paper axes, then outline another that follows the shoebox.
Both contain the object, but the axis-aligned outline includes more empty space and changes more when the shoebox rotates.
Arrow card for a safe approach
Tape three right-handed arrows to a cardboard gripper model and choose a separate point above a paper object as the pre-grasp location.
The object centre, tool arrival point, approach direction, and closing direction are distinct choices that must share one explicit convention.
Hands-on exercise
Make the idea observable
Use the fixed segmented clouds from Day 66: one rectangular box, one cylinder, and one deliberately partial view. Produce targets only as files or visualization markers.
Compute and overlay axis-aligned and oriented boxes for each candidate, recording dimensions, centre, point count, and the parent frame.
Rotate the rectangular cloud through several known angles and measure how AABB dimensions change and whether OBB axes stay consistent.
Estimate candidate object frames, then list equivalent orientations for the cylinder and flag any orientation component that the points cannot support.
Define approach and closing axes plus a pre-grasp offset, draw both object and tool frames, and assert right-handed rotation and unit quaternion norms where used.
Add rejection gates for stale input, too few points, excessive extent, poor support clearance, workspace limits, and unresolved orientation needed by the task.
Send a cancellation event through the offline pipeline and verify no new target is emitted and any queued candidate is marked invalid rather than executed.
AABB size should vary with rotation, a useful OBB should follow the rectangular object, and the cylinder should expose an orientation ambiguity instead of inventing a precise axis angle.
The saved overlay distinguishes bounds, object frame, pre-grasp frame, symmetry, and rejected candidates, and the cancellation trace ends without a usable target.
Build today
Transform an RGB-D observation into a filtered point cloud and safe grasp or approach target.
Evidence to save
DONE when the integrated “3D bounding boxes, 6D pose, and grasp approach frames” path is observable, cancelable, and leaves the prior baseline reproducible.
Common mistakes
Catch the wrong mental model
Using an oriented bounding box orientation as a guaranteed semantic object pose.
Treat it as a geometric estimate and validate axis stability, symmetry, partial-view bias, known object features, and task requirements.
Publishing a pose without parent frame, transform direction, timestamp, or quaternion validation.
Make the pose a complete stamped frame contract and assert rotation validity before any downstream planner reads it.
Choosing a grasp point at the object centre and assuming the tool can approach it.
Construct a task-specific approach frame and check pre-grasp path, gripper opening, object and support clearance, reachability, and collision geometry.
Job connection
How this becomes employable evidence
Define the typed boundary from a perception component to MoveIt or a policy service, including framed pose, dimensions, symmetry, quality, freshness, cancellation, and explicit rejection semantics.
Relevant target roles
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robot Learning Deployment / Physical AI Integration Engineer
Chapter 10 interview drill
Interview questions: 3D bounding boxes, 6D pose, and grasp approach frames
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Distinguish AABB, OBB, 6D object pose, and grasp approach frame, then explain how symmetry and a partial cloud can make a visually tidy pose unsafe to consume.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why can an AABB become larger when the physical object only rotates?
Its edges stay aligned to the parent-frame axes, so a diagonal object occupies a wider span along those fixed axes.
Q2Does 6D pose mean orientation must be stored as three Euler angles?
No. It means three positional and three rotational degrees of freedom; orientation can be represented by a valid rotation matrix or unit quaternion.
Q3What extra decision turns an object pose into an approach frame?
The task defines a tool origin and axis convention, including approach and closing directions, offsets, clearances, and acceptable symmetry alternatives.