Chapter 09 · Turn camera pixels into measured, debuggable robot observations
Today in the field story
One problem, then the next
Pass the Blue-Crate Inspection Cell frames through transparent stages: filter, threshold, edge, morphology, and contour. Save every intermediate image and change only one parameter while checking labelled small crates and noise. A clean-looking mask can erase a target, and a contour is merely a boundary in the current mask—not proof that a crate exists.
- Why now
Classical stages expose how assumptions turn pixel values into candidate regions.
- Ignore today
Ignore learned detection until the deterministic baseline and its errors are visible.
- Unlocks next
Candidate overlays and error evidence for calibration and detector comparison.
Understand
Build the physical picture first
An OpenCV pipeline is a row of transparent sieves: each stage keeps, removes, or reshapes image evidence according to assumptions that must stay visible.
A spatial filter computes an output pixel from a neighborhood around an input pixel. A 3 × 3 kernel examines nine locations; a larger kernel gathers evidence from farther away. Mean and Gaussian smoothing can reduce small fluctuations, while a median filter is useful against isolated bright or dark specks. Every smoother also removes some detail, so kernel shape, size, border handling, and input scale belong in the experiment record rather than being unexplained constants.
An edge detector responds to rapid intensity change, not to an object name. Image gradients estimate how quickly values change across x and y; Canny combines smoothing, gradients, non-maximum suppression, and two thresholds that connect strong and weak edge evidence. Texture, shadows, table seams, and glare can all create edges. A broken or closed edge can describe the same physical object under different lighting, so preserve the gradient or edge image beside the original.
Thresholding produces a binary mask, and contours trace connected boundaries in such a mask. A contour can provide area, perimeter, bounding box, and shape clues, but only after segmentation choices have decided foreground and background. Contour hierarchy matters when one boundary contains another, such as a dark ring around a bright center. Never call every contour an object; label which physical feature the mask is intended to represent and reject contours outside measured size or shape ranges.
Morphology edits binary regions with a structuring element. Erosion removes foreground near boundaries, dilation adds it, opening erodes then dilates to remove small foreground islands, and closing dilates then erodes to fill small holes or gaps. These operations are not generic cleanup buttons: a kernel larger than a real target can erase it, and closing can join two nearby objects. Compare intermediate masks against labelled images and count both removed noise and damaged targets.
Words you need
Name each idea precisely
- Kernel
A small neighborhood shape and set of weights or rules applied around each pixel.
Physical example:A 3 × 3 kernel examines a pixel and its eight immediate neighbors.
- Gradient
A numerical estimate of how quickly image intensity changes across position.
Physical example:The border between black paper and a white table produces a large brightness gradient.
- Binary mask
An image whose values mark accepted foreground and rejected background.
Physical example:White pixels represent a blue card selected by an HSV range; black pixels represent everything else.
- Contour
An ordered boundary around a connected region in a binary image.
Physical example:The outside edge of a thresholded paper circle becomes one contour whose area can be measured.
- Morphological opening
Erosion followed by dilation, normally used to remove foreground regions smaller than the structuring element.
Physical example:Single white specks disappear while a much larger white card region mostly keeps its shape.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalPixels, camera projection, and calibration error
A pixel becomes useful only after camera geometry and uncertainty are known.
- u, v
- pixel column and rowUnit: pixels (px)
- fₓ, fᵧ
- camera focal scaleUnit: pixels (px)
- Z
- depth along the camera axisUnit: metres (m)
Use x = (u − cₓ)Z/fₓ. Let u − cₓ = 100 px, Z = 2 m, fₓ = 500 px.
Multiply the numerator: 100 × 2 = 200 px·m.
Divide: x = 200/500 = 0.4 m; pixel units cancel, leaving metres.
Mobile camera pixels are familiar; robotics adds calibrated rays, a camera frame, and physical depth.
If u − cₓ = 50 px, Z = 1 m, and fₓ = 500 px, what is x?
x = 50×1/500 = 0.1 m.
A filter uses neighboring pixels. With , , and ,
These metrics compare pipelines; they do not choose the filter kernel.
Compare a cleaned mask with labelled target counts
A frozen set contains 20 real paper targets. After a 3 × 3 blur, color threshold, 3 × 3 opening, and contour-size filter, the pipeline reports 21 regions: 18 correctly match labelled targets and 3 are false regions; 2 labelled targets were missed.
Check the label count: 18 true positives + 2 false negatives = 20 real targets.
Check the prediction count: 18 true positives + 3 false positives = 21 reported regions.
Calculate precision as TP/(TP+FP) = 18/(18+3) = 18/21 ≈ 85.7%.
Calculate recall as TP/(TP+FN) = 18/(18+2) = 18/20 = 90%.
Inspect the three false-positive overlays to learn whether texture, glare, or remaining specks passed the size rule.
Inspect the two false-negative intermediate masks to see whether thresholding rejected them or morphology erased them; the counts alone cannot choose a new kernel.
The declared pipeline reaches about 85.7% precision and 90% recall on this frozen set, with five specific errors that must be traced through saved intermediate images.
Metrics compare an entire configured pipeline, while stage-by-stage masks reveal which assumption caused each error.
Physical examples
Where this appears in real life
Paper dots around a large square
A black paper square sits on white paper while tiny black confetti dots create isolated regions after thresholding.
A small opening can remove the dots, but an oversized kernel shrinks or removes narrow parts of the real square before dilation can restore them.
Broken ring under uneven light
A drawn circular ring is partly washed out by glare, so its binary contour contains a gap and may not be reported as one closed boundary.
Closing may bridge the gap, but a larger closing kernel can also connect the ring to a nearby shape and create one false combined region.
Hands-on exercise
Make the idea observable
Arrange printed circles, rectangles, one narrow shape, and several paper specks on a plain tabletop. Capture one non-sensitive image and keep it frozen while comparing pipeline variants.
Create labels for every intended shape, including the narrow target, before running OpenCV; record an allowed size range in pixels for this fixed camera pose.
Save the original, grayscale or HSV channel, blurred image, binary threshold mask, Canny edges, opened mask, closed mask, and final contour overlay with exact parameter names in each filename.
Run a 3 × 3 filter and structuring element, then tabulate which labelled shapes and paper specks remain after each stage.
Repeat with a 9 × 9 structuring element and record the first stage where the narrow target is damaged or removed.
Change one edge or threshold parameter at a time, never both, and explain which saved pixels changed and why the direction of change was expected.
Create a final error sheet listing every false region and missed target with the earliest pipeline stage that made recovery impossible.
Noise can disappear while target boundaries also move. The largest or smoothest-looking kernel is not automatically best, and contour output depends on all earlier segmentation decisions.
A reviewer can reproduce both kernel variants, inspect every intermediate image, reconcile TP/FP/FN counts with overlays, and identify the causal stage for each labelled error.
Build today
Detect, track, and estimate the pose of tabletop objects with an annotated evaluation set.
Evidence to save
DONE when “OpenCV filtering, edges, contours, and morphology” runs from one documented command and the nominal plus boundary outputs are attached.
Common mistakes
Catch the wrong mental model
Choosing a kernel because its final overlay looks clean on one favorite image.
Freeze representative labelled inputs, save every intermediate stage, and compare error counts plus smallest-target preservation across parameter variants.
Treating a Canny edge or contour as proof that an object was detected.
Edges mark intensity changes and contours bound mask regions; apply explicit physical size, shape, context, and validation rules before assigning object meaning.
Changing blur, threshold, morphology, and contour filters together after a failure.
Change one declared parameter at a time and use the earliest changed intermediate image to connect the repair to evidence.
Job connection
How this becomes employable evidence
An engineer debugging missed tote handles saves the live encoding, threshold mask, edge map, morphology output, contour filters, and configuration revision, then reproduces the miss offline and changes only the parameter whose intermediate image proves the failure.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 09 interview drill
Interview questions: OpenCV filtering, edges, contours, and morphology
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A colleague says a larger morphology kernel makes the mask cleaner. Explain erosion, dilation, opening, and closing, then design a labelled comparison that catches erased small targets, joined objects, and false contours.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why can a larger smoothing kernel reduce noise and also hurt detection?
It combines evidence over a wider neighborhood, which suppresses small variation but also spreads or removes narrow real details and moves boundaries.
Q2What does morphological opening do in its standard binary form?
It erodes foreground and then dilates it, usually removing foreground regions too small to survive the chosen structuring element.
Q3Why must contours be inspected with the mask that produced them?
A contour only traces a connected mask boundary; thresholding and morphology may already have deleted, split, filled, or joined physical regions.