Chapter 09 · Turn camera pixels into measured, debuggable robot observations
Today in the field story
One problem, then the next
Two blue crates cross inside the Blue-Crate Inspection Cell and the tracker swaps their identities despite detecting both. Record detection, prediction, observed age, association score, track ID, occlusion, missed updates, and deletion state separately. Replay the crossing, exit, re-entry, and false-positive sequences so smooth boxes cannot conceal an ID switch or stale predicted observation.
- Why now
A robot task often needs persistent identity, not only accurate boxes in isolated frames.
- Ignore today
Ignore complex multi-camera tracking and appearance-model training.
- Unlocks next
A labelled temporal observation stream and explicit identity-failure taxonomy.
Understand
Build the physical picture first
Detection answers what appears in one frame; tracking carries a cautious identity story across time and must label when that story is only a prediction.
A tracker maintains state across ordered frames. At minimum, each track needs a track ID, latest region or position, source timestamp, update status, and age since reliable observation. A detector can initialize or refresh that state, while a motion or appearance model can predict between detections. The track ID is a software handle, not a property visible on most objects. Restarting the tracker or making a bad association can change identity even when every frame contains a correct box.
Data association decides which new detection belongs to which existing track. Simple baselines compare box IoU or center distance; stronger systems may also use predicted motion and appearance features. Crossing paths create ambiguity because both objects can be close to either previous box. A greedy highest-IoU choice can work in easy motion and still swap IDs after overlap. Record the full candidate costs and chosen match rather than drawing a smooth line that hides the decision.
Occlusion means another surface blocks the camera's view. During a short occlusion, a tracker may predict a location, but it must mark that state predicted or lost and increase its age or uncertainty. Confirmation rules can require several detections before a track is born; deletion rules retire tracks after a declared missed-update limit. Reappearing objects may keep or receive a new ID according to a documented re-identification rule. Never publish a predicted box as a fresh camera observation.
Tracking labels extend beyond frame-level TP, FP, and FN. They must preserve object identity through time so evaluation can count ID switches, track fragmentation, premature deletion, and false persistence after an object leaves. Perfect detection recall is compatible with wrong tracking when every object is boxed but identities swap. Use timestamps and recorded sequences, replay the same fault cases, and require downstream robot logic to reject stale or identity-ambiguous tracks.
Words you need
Name each idea precisely
- Track ID
A software identifier intended to refer to one object history across frames.
Physical example:The left token begins as track 4 and should remain track 4 after moving right unless identity becomes unresolved.
- Data association
The decision that pairs current detections with existing tracks or starts new tracks.
Physical example:Two new boxes are compared with the predicted boxes for tracks 4 and 7 before matches are assigned.
- Occlusion
A viewing condition where another surface partly or fully hides the target.
Physical example:A paper token disappears for two frames while passing behind an upright card.
- ID switch
An error where the reported identity assigned to a physical object changes or swaps with another object's identity.
Physical example:After two tokens cross, the red token receives the track ID that previously followed the blue token.
- Track age
Elapsed time or frame count since a track was created or last supported by a reliable observation, according to the declared field.
Physical example:A predicted location is 120 ms old because the object has not been detected since the earlier timestamp.
- Track fragmentation
One physical object's path being split into multiple separate reported tracks.
Physical example:Token A is track 3 before a brief occlusion and a new track 9 after it returns.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalPixels, camera projection, and calibration error
A pixel becomes useful only after camera geometry and uncertainty are known.
- u, v
- pixel column and rowUnit: pixels (px)
- fₓ, fᵧ
- camera focal scaleUnit: pixels (px)
- Z
- depth along the camera axisUnit: metres (m)
Use x = (u − cₓ)Z/fₓ. Let u − cₓ = 100 px, Z = 2 m, fₓ = 500 px.
Multiply the numerator: 100 × 2 = 200 px·m.
Divide: x = 200/500 = 0.4 m; pixel units cancel, leaving metres.
Mobile camera pixels are familiar; robotics adds calibrated rays, a camera frame, and physical depth.
If u − cₓ = 50 px, Z = 1 m, and fₓ = 500 px, what is x?
x = 50×1/500 = 0.1 m.
Intersection over union is
That value may associate two boxes. Detector precision and recall score , , and , but cannot reveal an identity switch when both boxes remain detected.
Make one association, then expose an identity switch
Track 4 has a predicted box in the next frame. Detection A overlaps it by 60 pixels with a union of 100 pixels; Detection B overlaps it by 20 pixels with the same 100-pixel union. The association gate is IoU ≥ 0.50. One frame later, two similar tokens cross and a manual identity label shows the chosen track now follows the other token.
Calculate IoU(track 4, A) = 60/100 = 0.60.
Calculate IoU(track 4, B) = 20/100 = 0.20.
Under the stated gate, A is eligible and B is not, so assign A to track 4 for this frame.
Keep the candidate values and source timestamp; the 0.60 match is a rule-based association, not proof of physical identity.
At the crossing frame, compare the reported track with the manual token identity and mark the first wrong assignment as an ID switch.
Report detection and tracking separately: boxes may remain true positives while the identity history fails from the switch onward.
Detection A passes the 0.50 association gate with IoU 0.60, but the later labelled crossing proves that the track's identity still switches.
Association scores select a hypothesis under a model; identity labels over time are needed to test whether that hypothesis remained true.
Physical examples
Where this appears in real life
Two tokens crossing behind a card
A red coin and a blue coin move toward each other, overlap near an upright card, disappear briefly, and emerge on opposite sides.
Mark every detection, predicted state, lost interval, and ID. Correct boxes after the crossing do not prove that identity was preserved.
Package leaving a conveyor view
A cardboard package moves out of the camera frame while a textured background still resembles part of its last box.
The lifecycle should age and delete the track under a declared rule; a box that follows background texture is false persistence, not object survival.
Hands-on exercise
Make the idea observable
Use two visually distinct paper tokens, an upright card for occlusion, a phone or webcam, and either an OpenCV single-object tracker or a simple detection-plus-IoU association script. Keep the tabletop stationary.
Record a short fixed-camera sequence where both tokens move, cross, one is hidden for two or three frames, and one finally leaves the image.
Label each visible token with the same ground-truth identity and box in every frame; mark fully occluded intervals and true exit separately.
Initialize tracks from the first frame, then log for every later frame the source timestamp, candidate detections, association scores, chosen match, track status, and time since last reliable observation.
Use an IoU gate as the first baseline, replay the sequence, and annotate every missed detection, false detection, ID switch, fragmentation, and false persistence event.
Add a declared predicted/lost state plus a maximum missed-update or age limit; ensure predicted boxes are visually and structurally different from fresh detections.
Replay exactly the same frames, compare the error table with the baseline, and retain one crossing or occlusion case that still fails rather than hiding it.
Export a ten-frame strip showing labels, track IDs, observed versus predicted status, and the first causal association decision for each identity error.
A tracker can draw smooth boxes through an occlusion while its identity is wrong or its observation is stale. Exit, hidden, missed, and predicted are different states with different robot consequences.
The replay is deterministic, every track state links to a timestamp and association decision, observed and predicted boxes cannot be confused, and the report counts identity errors in addition to frame detections.
Build today
Detect, track, and estimate the pose of tabletop objects with an annotated evaluation set.
Evidence to save
DONE when a 60–120 second uncut “Tracking, occlusion, false positives, and data labeling” demo links to its command, logs or plots, result count, and honest failure note.
Common mistakes
Catch the wrong mental model
Using the class label, such as box, as if it uniquely identifies one physical object.
Maintain a separate track ID and validate that identity over time with labelled sequences, especially at crossings and occlusions.
Drawing a predicted box with the same status and timestamp semantics as a detector-supported observation.
Publish observed, predicted, lost, and deleted states explicitly with last-observation time and a hard freshness or uncertainty limit.
Reporting only frame-level precision and recall for a tracking system.
Also count ID switches, fragmentation, false persistence, lifecycle timing, and errors on crossing, occlusion, entry, and exit slices.
Deleting a track immediately after one missed detection or preserving it forever after exit.
Define confirmation, missed-update, age, and deletion rules from the physical motion and downstream safety contract, then replay boundary cases.
Job connection
How this becomes employable evidence
When two totes cross near an occluding shelf, the engineer replays timestamped detections and association costs, finds the first ID switch, bounds predicted-track age, and prevents the motion layer from treating a stale or ambiguous identity as a fresh target.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robot Learning Deployment / Physical AI Integration Engineer
Chapter 09 interview drill
Interview questions: Tracking, occlusion, false positives, and data labeling
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Detection recall is 100%, but a robot follows the wrong package after two packages cross. Explain association, IoU limits, motion and appearance cues, ID switches, occlusion states, lifecycle rules, timestamps, and the acceptance evidence you need.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1How can detection recall be 100% while tracking identity is wrong?
Every object can have a correct box in every visible frame while the tracker assigns those boxes to the wrong persistent IDs.
Q2What should a tracker publish during a short full occlusion?
If it predicts, it should publish an explicit predicted or lost state with last reliable observation time and bounded age or uncertainty, not claim a fresh detection.
Q3What does IoU = 0.60 prove about an association?
Only that the two boxes overlap by 60% of their union; under a chosen gate it may be an eligible match, but it does not prove physical identity.