Chapter 11 · Estimate robot state, localize honestly, and measure uncertainty

Today in the field story

One problem, then the next

Replay the complete blackout aisle: nominal loop, IMU outlier, missing lidar, accumulated drift, landmark return, and recovery. Correlate every estimate with ground truth, covariance, transform age, configuration, and event time. Report when uncertainty covered the actual error and when it did not. The final decision must name the estimator’s useful envelope and the exact dropout or geometry that still blocks acceptance.

Why now: Separate estimator ideas become engineering evidence only in one frozen failure-and-recovery run.
Ignore today: Ignore field deployment claims; qualify the recorded simulation mission and its known limits.
Unlocks next: Localization evidence reusable by navigation, FleetOps, and later validation ladders.

Understand

Build the physical picture first

A localization report is a weather forecast scorecard: it compares the estimate with independent truth, checks whether confidence was honest, and records what happened when observations disappeared.

Start with a frozen evaluation question and reference. Simulation ground truth, motion capture, surveyed fiducials, or another documented method can supply pose truth, but estimate and truth must share frame, time, angle convention, and sample policy. Preserve scenarios, seeds, route, speeds, sensor settings, software versions, warm-up exclusions, and all failed runs before computing aggregate numbers.

Measure more than one average. Position RMSE penalizes large errors, mean absolute error is easier to interpret, maximum error exposes the worst observed miss, and heading error needs angle wrap handling. Also record drift versus distance, transform age, update latency, localization-loss count, and time outside the task tolerance. Break results down by nominal, repeated geometry, loop closure, outlier, and dropout conditions.

Uncertainty needs calibration evidence. Covariance is an internal claim; coverage asks how often independent truth falls inside the claimed interval or region. Too little coverage signals overconfidence, while extremely broad intervals can achieve high coverage without being useful. Inspect normalized innovations or another appropriate consistency statistic when available, but state its model assumptions and do not replace pose-error measurement with it.

A dropout trial has explicit phases: healthy baseline, observation removal, prediction-only period, observation return, validation, and either recovered or failed terminal state. Define detection time, allowed behavior while uncertain, maximum error, maximum uncertainty, recovery threshold, required sustained duration, jump limit, and retry or stop policy before the run. The first fresh scan is not recovery if pose remains wrong or unstable.

The final package connects configuration to raw evidence and decision. Include architecture and frame ownership, sensors and calibration, process and measurement models, Q and R settings, initialization, map provenance, trial manifest, metrics with denominators, uncertainty plots, dropout timelines, bags, exact commands, failure taxonomy, limitations, and a release verdict. A neat map or successful demo is supporting evidence, never the complete claim.

Words you need

Name each idea precisely

Localization error: The pose difference between estimate and an independent reference after frame and time alignment.
Physical example:
Estimated x-y position is 0.12 m from Gazebo truth at the same timestamp.
RMSE: The square root of mean squared error, which gives larger misses extra influence.
Physical example:
Position errors 0.05, 0.12, and 0.07 m produce RMSE about 0.085 m.
Coverage: The fraction of independent truth samples contained by a stated uncertainty interval or region.
Physical example:
Only 80 of 100 truth poses fall inside regions claimed to contain 95%.
Sensor dropout: A defined interval in which an expected observation source provides no usable updates.
Physical example:
Lidar messages are removed from 10.0 s through 13.0 s during bag replay.
Recovery time: Elapsed time from the declared recovery start event until all recovery criteria hold for the required duration.
Physical example:
Scans return at 13.0 s and error stays below 0.10 m from 14.2 s onward, giving 1.2 s recovery.
Failure taxonomy: A stable set of actionable cause categories used to count and investigate failed trials.
Physical example:
Failures are separated into timing, transform, sensor, matching, map, initialization, and recovery-policy causes.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalProbability, variance, and Kalman weighting

State estimation combines predictions and measurements according to uncertainty.

μ: best current estimateUnit: state unit
σ²: variance, or squared uncertainty spreadUnit: state unit squared
K: Kalman gain, the measurement weightUnit: unitless

Prediction is 10 m. Measurement is 12 m. Let K = 0.25.
Innovation is 12 − 10 = 2 m.
Updated estimate = 10 + 0.25×2 = 10.5 m; the lower-trust measurement only shifts the estimate partway.

Programmer analogy

It is a weighted merge like resolving two data sources, but the weights come from modeled uncertainty.

Prediction 5 m, measurement 7 m, K = 0.5. What is the update?

5 + 0.5×(7−5) = 6 m.

For errors $0.05$ , $0.12$ , and $0.07\ \mathrm{m}$ ,

\operatorname{RMSE}=\sqrt{\frac{0.05^2+0.12^2+0.07^2}{3}}\approx0.085\ \mathrm{m}.

The maximum observed error is separately $0.12\ \mathrm{m}$ .

Score accuracy, uncertainty, and dropout recovery separately

Three aligned position-error magnitudes are 0.05, 0.12, and 0.07 m. In a ten-sample teaching set, only eight truths lie inside the claimed 95% regions. Lidar returns at 13.0 s and error first remains below the 0.10 m recovery bound at 14.2 s.

Square the three errors: 0.05² = 0.0025, 0.12² = 0.0144, and 0.07² = 0.0049 m².
Average the squares: (0.0025 + 0.0144 + 0.0049) / 3 = 0.0072667 m².
Take the square root: RMSE = √0.0072667 ≈ 0.0852 m.
Report maximum observed error separately as 0.12 m; RMSE does not preserve which sample was worst.
Calculate teaching-set coverage: 8 / 10 = 80%, which is below the claimed 95%; ten samples are also too few for a strong production calibration claim.
Calculate recovery time from observation return: 14.2 - 13.0 = 1.2 s, then verify the error stayed within 0.10 m for the predeclared sustained interval.

Result

The run has 0.085 m RMSE, 0.12 m maximum error, only 80% observed coverage for a 95% claim, and 1.2 s recovery after lidar returns.

What this proves

Accuracy, worst case, uncertainty honesty, and recovery answer different questions; none should be allowed to stand in for the others.

Physical examples

Where this appears in real life

Forecast interval scorecard

A week of weather forecasts gives a temperature range each day, and actual temperatures are later marked against those ranges.

Look for:

Narrow ranges that frequently miss are overconfident; extremely wide ranges may cover every day but provide little decision value.

Covered trail with measured checkpoints

A toy cart travels behind a cardboard screen using wheel marks, loses occasional visible checkpoints, then reappears beside ruler-measured reference marks.

Look for:

Prediction continues while hidden, uncertainty should grow, and recovery is judged by sustained agreement with reference marks rather than the first sighting.

Hands-on exercise

Make the idea observable

Reopen the exact Week 8 source/container identity, Gazebo world and seed, MCAP bag, map image/YAML, serialized pose graph, launch/config/BT files, TF snapshot, scenario manifest, and raw results. Verify their recorded hashes before using separate simulation truth.

Write acceptance criteria and hard gates for frame and transform ownership, nominal position and heading error, covariance coverage, dropout detection, allowed prediction-only behavior, recovery time, jump size, and terminal state.
Run the nominal seed set and save aligned estimate, covariance, truth, innovations or residuals, transform age, update latency, map or particle evidence, and complete denominators.
Replay the same set with a fixed lidar dropout plus one bounded outlier or timestamp fault, retaining runs that fail initialization or never recover.
Calculate per-run and scenario-level RMSE, mean absolute and maximum error, heading error, time outside tolerance, coverage, loss count, detection time, recovery time, and operator or stop events.
Inspect the worst run and one overconfident run against raw messages, transforms, diagnostics, map, and configuration; assign an actionable cause rather than an outcome-only label.
Package commands, versions, unchanged Week 8 artifact hashes, configs, map provenance, seed manifest, raw CSV, plots, bags, calculations, failures, limitations, and a PASS, NEEDS REVISION, or BLOCKED verdict tied to the frozen criteria.

Observe

Prediction-only intervals widen uncertainty and accumulate error; observation return may correct pose, oscillate, jump, or fail, and aggregate averages can hide those distinct outcomes.

Done when

A reviewer can rerun the exact suite, recalculate every headline number from raw rows, inspect the worst dropout trace, and reach the same evidence-limited verdict.

Build today

Use the frozen Week 8 artifacts to fuse odometry and IMU, explain scan matching, loop closure, and AMCL, then rerun the unchanged scenarios and compare error, uncertainty coverage, transform age, dropout recovery, and navigation success.

Evidence to save

DONE when the weekly ship note explains how “SLAM/localization report with uncertainty and dropout recovery” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Reporting only average or RMSE localization error.

Better

Add maximum and percentile error, heading, scenario splits, time outside tolerance, loss events, uncertainty coverage, and recovery metrics.

Wrong

Using estimator covariance as the ground-truth error curve.

Better

Plot covariance as the estimator's claim and calculate error independently from a reference aligned in frame and time.

Wrong

Declaring recovery on the first observation after dropout.

Better

Require fresh data plus sustained error, uncertainty, transform, jump, and system-state criteria for a predeclared duration.

Wrong

Removing failed initialization or non-recovery trials from the denominator.

Better

Retain every planned trial, mark predeclared exclusions explicitly, and report actionable failure categories beside aggregate metrics.

Job connection

How this becomes employable evidence

Own the localization release report for an AMR deployment, including calibrated uncertainty checks, fixed dropout and outlier replay, recovery hard gates, traceable failures, and evidence-backed release judgment.

Relevant target roles

Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 11 interview drill

Interview questions: SLAM/localization report with uncertainty and dropout recovery

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Design an estimator acceptance test that distinguishes low average error, worst-case error, overconfidence, sensor-dropout behavior, false recovery, and missing evidence before a customer-site release.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What is the difference between covariance and localization error?

Model interview answer

Covariance is the estimator's uncertainty claim; localization error is measured against an independent reference in common frame and time.

Q2When should dropout recovery time start and end?

Model interview answer

Start at the declared event, commonly valid observation return, and end only when all recovery criteria hold for the required sustained interval.

Q3Why report maximum error as well as RMSE?

Model interview answer

RMSE combines samples and can hide the single worst miss that may violate a task or safety tolerance.

Chapter starter artifact

Update one belief and reject stale evidence

A replayable localization report follows Courier-3 through nominal motion, an IMU outlier, lidar dropout, drift, loop closure, and recovery, reporting position and heading error, uncertainty coverage, transform freshness, recovery time, configuration, raw evidence, and a bounded acceptance decision.

week-11-courier-belief-update.mjsJavaScriptDownload starter

const prior = { mean: 2.0, variance: 1.0 };
const controlDelta = 1.0;
const processVariance = 1.0;
const maxAgeMs = 100;
const measurements = [
  { value: 4.0, variance: 0.5, ageMs: 40 },
  { value: 9.0, variance: 0.5, ageMs: 180 },
  { value: 5.0, variance: 0.5, ageMs: -10 },
  { value: Number.NaN, variance: 0.5, ageMs: 20 },
];
const isValid = (item) =>
  [item.value, item.variance, item.ageMs].every(Number.isFinite) &&
  item.variance > 0 && item.ageMs >= 0;
const valid = measurements.filter(isValid);
const fresh = valid.filter((item) => item.ageMs <= maxAgeMs);
const stale = valid.filter((item) => item.ageMs > maxAgeMs).length;
const invalid = measurements.length - valid.length;
const predictedMean = prior.mean + controlDelta;
const predictedVariance = prior.variance + processVariance;
const measurement = fresh[0];
if (!measurement) throw new Error("no valid fresh measurement");
const gain = predictedVariance / (predictedVariance + measurement.variance);
const posteriorMean =
  predictedMean + gain * (measurement.value - predictedMean);
const posteriorVariance = (1 - gain) * predictedVariance;
const output =
  "fresh=" + fresh.length +
  " stale=" + stale +
  " invalid=" + invalid +
  " gain=" + gain.toFixed(3) +
  " posterior=" + posteriorMean.toFixed(2) +
  " variance=" + posteriorVariance.toFixed(3);
const expected = "fresh=1 stale=1 invalid=2 gain=0.800 posterior=3.80 variance=0.400";
if (output !== expected) throw new Error("belief mismatch: " + output);
console.log(output);

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-11-courier-belief-update.mjs
Expected output: fresh=1 stale=1 invalid=2 gain=0.800 posterior=3.80 variance=0.400
Planted failure to diagnose: The 180 ms measurement is stale, the negative age is future-dated, and the NaN value is not evidence; coercive comparisons or finite checks applied to only one field admit invalid observations.