Chapter 10 · Turn depth measurements into safe, inspectable 3D targets

Today in the field story

One problem, then the next

All parts now converge on one replayable tote decision. Run raw depth, mask, cloud, plane removal, target extraction, calibration, pose, and approach-frame checks under one correlation identity. The final overlay must show accepted evidence and rejected regions without hiding the raw values. Publish counts, timings, residuals, and the exact planted failures so the MoveIt chapter receives a measured target contract rather than a screenshot.

Why now: The chapter closes only when separate geometry exercises form one reproducible decision path.
Ignore today: Ignore powered motion and model training; prove perception and rejection on frozen evidence.
Unlocks next: A framed, quality-gated target ready for whole-arm collision and planning tests.

Understand

Build the physical picture first

A trustworthy 3D target travels through a chain of evidence gates, and every gate must show both what passed and why a bad case stopped.

The weekly pipeline begins with synchronized colour or target region, raw depth, CameraInfo, frame identifiers, and acquisition stamps. It verifies encoding, dimensions, calibration version, scale, finite values, range, and valid coverage before back-projecting. It then transforms points at the measurement time, crops the allowed workspace, downsamples at a declared voxel size, and records counts and bounds after each stage. If a required transform or metadata contract is missing, the pipeline ends with a reason rather than guessing.

Geometry stages remove or label the support plane, cluster candidate object points, compute bounds, and estimate a pose only when the task needs one. The approach-frame stage applies object and gripper conventions, symmetry choices, offsets, workspace limits, and support clearance. Each output carries its source stamp, parent frame, dimensions, quality values, and valid-until rule. The result is still a candidate for a planner or reviewer, never direct proof that physical motion is safe.

Visual overlays are diagnostic evidence. On the colour image, show the target region, valid-depth coverage, projected centre, and rejection label. In 3D, show the raw or filtered cloud, removed plane, candidate cluster, object axes, approach axes, and excluded points in different labelled colours. A screenshot without raw counts, transform direction, configuration identity, and timestamp is not reproducible. Link each overlay to the exact run and result row.

Evaluate a frozen set containing ordinary objects and deliberate bad cases: partial occlusion, reflective or transparent material, sparse depth, wrong scale, stale transform, reversed extrinsic, oversized voxel, bad plane threshold, poor registration seed, symmetry, and out-of-workspace target. Report acceptance, false accept, false reject, position and orientation error where references exist, valid coverage, latency distribution, and rejection reasons. A good release can safely reject impossible inputs; it must not hide them to improve success rate.

The final ship note should separate verified facts from remaining risk. Include one command, versions, source data, configuration, stage metrics, result table, overlays, logs, and limitations. Explain the highest-impact failure and the smallest next experiment that could disprove your current belief. This produces evidence relevant to integration, validation, robot software, operator tooling, and physical-AI deployment without pretending that an offline RGB-D exercise validates a powered robot.

Words you need

Name each idea precisely

Quality gate: A declared, measurable condition that a stage must pass before its output may be consumed downstream.
Physical example:
Pose estimation is skipped when valid depth covers less than 92% of the target region.
Stage metric: A value recorded at one pipeline boundary so loss, delay, and rejection can be traced.
Physical example:
The transform stage records input count, output count, frame, transform age, and processing milliseconds.
Structured rejection: A machine-readable failure result containing a specific reason and the evidence that violated a gate.
Physical example:
STALE_TRANSFORM reports the measurement stamp, transform stamp, observed age, and allowed age.
Position error: The Euclidean distance between an estimated 3D position and a trusted reference in the same frame and time.
Physical example:
Errors of 0.03 m in x and 0.04 m in y combine into 0.05 m position error.
Bad-case set: A fixed collection of realistic failure inputs retained alongside nominal examples for repeatable evaluation.
Physical example:
The set includes glass, foil, an occluded box, wrong depth scale, stale transform, and a target outside the workspace.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalDepth, point distance, and rigid alignment

3D perception needs distances and poses that agree across frames.

p = [x,y,z]: a 3D pointUnit: metres (m)
||p||: distance from the frame originUnit: metres (m)
eᵢ: alignment residual for point iUnit: metres (m)

For p = [0.3, 0.4, 0] m, square the components: 0.09 + 0.16 + 0.
The sum is 0.25 m².
Distance is √0.25 = 0.5 m; invalid or missing depth must be excluded before this calculation.

Programmer analogy

Like mapping API objects between schemas, point clouds require an explicit transform; unlike JSON, a wrong transform can still look plausible.

How far is [0, 0, 2] m from the camera origin?

2 m.

For coordinate errors $[0.03,0.04,0]\,\mathrm{m}$ , position error is

e=\sqrt{(\Delta x)^2+(\Delta y)^2+(\Delta z)^2}=\sqrt{0.03^2+0.04^2}=0.05\,\mathrm{m}.

A coordinate may be back-projected as $X=(420-320)\times2/500=0.4\,\mathrm{m}$ . Neither value proves the whole pipeline is robust.

Reject a target that misses the task-space accuracy gate

A frozen test target has reference centre (0.50, 0.20, 0.80) m in base_link. The pipeline estimates (0.53, 0.24, 0.80) m, reports 95% valid depth, 72 ms latency, and all metadata checks pass. The release gates require position error at most 0.04 m, coverage at least 92%, and latency at most 100 ms.

Place estimate and reference in the same frame and timestamp contract before subtracting them.
Compute coordinate errors: (0.03, 0.04, 0.00) m.
Compute position error: sqrt(0.03² + 0.04² + 0²) = 0.05 m.
Check other gates: 95% ≥ 92% coverage passes, and 72 ms ≤ 100 ms latency passes.
Reject the target because 0.05 m > 0.04 m; two passing gates cannot override the failed accuracy hard gate.
Return POSITION_ERROR_LIMIT with the measured error, threshold, frame, run identity, overlay, and stage metrics for causal investigation.

Result

The target is rejected despite good coverage and latency because its 5 cm reference error exceeds the declared 4 cm task limit.

What this proves

An end-to-end verdict is the conjunction of required gates, not an average score that allows one unsafe geometric error to disappear.

Physical examples

Where this appears in real life

Airport security trays

Items move through several staffed stations, and a tray stops with a labelled reason when identity, scan, size, or routing evidence is missing.

Look for:

A later station never invents a pass for a failed earlier check, and the tray identity connects every observation and decision.

Transparent tracing stack

Layer separate sheets for camera pixels, valid depth, camera points, base points, table removal, object bounds, and approach axes.

Look for:

Removing or shifting one sheet reveals which downstream geometry changes and which evidence must remain linked to the same run.

Hands-on exercise

Make the idea observable

Assemble the week's offline or recorded stages into one reproducible command and freeze at least one nominal target plus three distinct bad cases.

Validate synchronized inputs, camera metadata, depth scale, validity, frame names, stamps, calibration identities, and required transforms before creating any 3D point.
Run back-projection, crop, voxel filtering, frame transformation, support-plane extraction, clustering, bounds, pose, and approach-frame construction with per-stage counts and timing.
Generate linked 2D and 3D overlays showing measured support, rejected pixels or points, plane, candidate, object axes, approach axes, quality values, and terminal reason.
Execute the frozen nominal and bad-case set, retaining every row with expected status, observed status, coverage, errors where referenced, latency, and rejection reason.
Repair the highest-impact false accept or silent failure, add its exact input as a regression case, then rerun the entire set rather than only the repaired example.
Package the command, versions, inputs, configs, calibration record, raw result table, overlays, logs, limitations, and next falsifiable experiment in the ship note.

Observe

Every final candidate or rejection should trace to one input identity and consistent frame-time chain, and the repaired pipeline should preserve nominal behavior while blocking its recorded bad case.

Done when

A clean reviewer reproduces the full set, recalculates one error from raw values, follows one rejection through all stages, and reaches the same offline release verdict.

Build today

Transform an RGB-D observation into a filtered point cloud and safe grasp or approach target.

Evidence to save

DONE when the weekly ship note explains how “3D target pipeline with visual overlays and bad cases” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Displaying only the final green box and calling the pipeline validated.

Better

Link the box to raw inputs, metadata, masks, transform chain, stage counts, configuration, metrics, bad cases, and a reference-based verdict.

Wrong

Allowing downstream stages to run after an upstream metadata or quality gate fails.

Better

Stop the candidate path, emit a structured reason, and test that no stale or fabricated target remains available to a consumer.

Wrong

Reporting average position error and average latency without individual failures or denominators.

Better

Keep every trial row, distribution or percentile timing, worst cases, false accepts, false rejects, and hard-gate violations visible.

Wrong

Claiming the offline point-cloud lab proves grasping safety on hardware.

Better

Claim only the recorded perception behavior; physical commissioning still needs robot calibration, planning, collision, load, stopping, supervision, and safety validation.

Job connection

How this becomes employable evidence

Own a perception release boundary from ROS sensor contracts through 3D geometry to planner, policy, or operator consumers, with traceable overlays, latency, accuracy, bad-case regressions, cancellation, and structured rejection.

Relevant target roles

Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robot HMI / Control & Monitoring Engineer
Robot Learning Deployment / Physical AI Integration Engineer

Chapter 10 interview drill

Interview questions: 3D target pipeline with visual overlays and bad cases

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Walk through your RGB-D target pipeline stage by stage, name every frame and gate, show a false accept you repaired, and state exactly what the offline evidence does and does not prove about a real robot.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What should happen when CameraInfo, image dimensions, or a required timestamped transform is inconsistent?

Model interview answer

The pipeline should stop target production and return a specific metadata or transform rejection rather than guessing a calibration or using the latest convenient transform.

Q2Why include both 2D and 3D overlays?

Model interview answer

The image overlay exposes pixel support and missing depth, while the 3D overlay exposes frame transforms, segmentation, bounds, pose, approach axes, and excluded geometry.

Q3What can this completed week honestly prove?

Model interview answer

It can prove reproducible offline or simulated sensor-to-target behavior and rejection under the frozen cases; it cannot prove real-robot motion, grasp, collision, stopping, or production safety.

Chapter starter artifact

Reject invalid depth before averaging

A versioned offline 3D-target report identifies the tote in a named frame, shows synchronized overlays and residuals, rejects the glossy and occluded bad cases with explicit reasons, and links every accepted pose to the exact depth fixture, calibration identity, command, and result table.

week-10-depth-quality-gate.mjsJavaScriptDownload starter

const readings = [
  { pixel: "320,240", depthM: 1.0, valid: true },
  { pixel: "322,240", depthM: 0.0, valid: false },
  { pixel: "324,240", depthM: 1.5, valid: true },
  { pixel: "326,240", depthM: 4.8, valid: true },
  { pixel: "328,240", depthM: 1.1, valid: "false" },
  { pixel: "", depthM: 1.2, valid: true },
];
const limits = { minM: 0.2, maxM: 2.0 };
const accepted = readings.filter(
  (sample) =>
    typeof sample.pixel === "string" &&
    sample.pixel.length > 0 &&
    sample.valid === true &&
    Number.isFinite(sample.depthM) &&
    sample.depthM >= limits.minM &&
    sample.depthM <= limits.maxM,
);
const rejected = readings.length - accepted.length;
const meanDepthM =
  accepted.reduce((sum, sample) => sum + sample.depthM, 0) / accepted.length;
const output =
  "accepted=" + accepted.length +
  " rejected=" + rejected +
  " meanDepthM=" + meanDepthM.toFixed(2);
const expected = "accepted=2 rejected=4 meanDepthM=1.25";
if (output !== expected) throw new Error("depth gate mismatch: " + output);
console.log(output);

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-10-depth-quality-gate.mjs
Expected output: accepted=2 rejected=4 meanDepthM=1.25
Planted failure to diagnose: The 4.8 m return is outside the shelf envelope, while string false and a missing pixel identity are malformed evidence; truthiness or range-only filtering accepts them incorrectly.