Chapter 21 · Bound embodied agents with safety and release evidence

Today in the field story

One problem, then the next

The evaluator refuses one blended success score for Courier-17. Each scenario receives one terminal label such as success, partial success, safe abort, unsafe outcome, or invalid test, while guard rejection, intervention, recovery, latency, stale state, and unsafe proposal remain separate event counters. Denominators stay visible by condition. Human-factors notes also record misunderstood states, handoff delay, workload, and intervention confidence. A high task-completion average can no longer hide repeated human rescue, unusable warnings, or a single prohibited event.

Why now: Metrics shape what failures the team can see and act upon.
Ignore today: Ignore composite leaderboards and post-hoc threshold changes.
Unlocks next: An outcome schema that supports defensible scenario aggregation.

Understand

Build the physical picture first

A trustworthy scorecard keeps terminal outcomes exclusive while counting interventions, guard actions, timing, and hazards on their own denominators.

Start with mutually exclusive terminal outcomes so every trial contributes exactly once: autonomous success, assisted completion, partial completion, safe abort, or unsafe physical outcome. Define each category before testing. A task that finishes after an operator moves an object is assisted, not autonomous; a robot that refuses an ambiguous command may be a correct safe abort, not task success; a late unsafe contact remains unsafe even if the object reaches its target.

Count cross-cutting events separately because they can occur inside any terminal trial. Intervention requests, actual human interventions, clarifications, guard rejections, unsafe proposals, controller rejections, retries, recovery attempts, deadline misses, and protective stops may overlap. Give each metric a denominator that matches its question: interventions per trial, unsafe proposals per proposal, recovery successes per eligible failure, and latency percentiles over requests with a defined start and end.

An aggregate can conceal the conditions that matter most. Slice outcomes by scenario family, object, lighting, prompt form, initial pose, robot embodiment, fault type, model and ruleset version, and operator mode. Report the worst relevant slice and raw counts beside percentages. Thirty trials are enough for a transparent learning exercise, not for a universal reliability claim; avoid false precision and retain every trace that determines a count.

Safety and utility metrics should be read together, not merged. Zero unsafe physical events in a small frozen suite does not prove future safety, while many guard rejections can indicate either effective containment or a dangerously poor upstream policy. A release gate can require zero forbidden events, complete trace coverage, and minimum utility and recovery thresholds, but the decision must remain scoped to the exact build, scenarios, environment, and deployment stage evaluated.

Words you need

Name each idea precisely

Terminal outcome: The single mutually exclusive state assigned when one trial ends under its declared task and safety predicates.
Physical example:
A blocked route that stops and requests help is one safe-abort trial, even if it also contains a clarification and a guard rejection.
Autonomous success: Completion of the predeclared task predicate without human assistance and without any forbidden event.
Physical example:
The robot identifies, picks, places, and verifies the correct parcel while the operator only observes.
Assisted completion: Task completion that required a human action beyond the assistance allowed by the autonomous-success definition.
Physical example:
An operator selects one of two identical boxes, after which the robot finishes the placement.
Safe abort: A controlled terminal state that prevents continued execution when completion or recovery cannot proceed within the declared envelope.
Physical example:
After two failed re-observations, the arm cancels, drains queued commands, holds a defined state, and requests help.
Intervention rate: The number of trials with actual human intervention divided by all in-scope trials, with the intervention definition stated.
Physical example:
Five of thirty trials need a person to clarify, move an object, or recover the system, so the trial intervention rate is 16.7%.
Containment metric: A count of hazardous or unauthorized proposals that were blocked before the prohibited physical effect, reported separately from proposal quality.
Physical example:
Seven excessive-speed proposals are all rejected, yielding seven contained proposals and zero resulting speed violations in the frozen run.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalRisk, stopping distance, and confidence

Safety claims need physical bounds and enough trials, not reassuring language.

d_react = vt: distance travelled during reaction delayUnit: metres (m)
d_brake = v²/(2a): ideal braking distanceUnit: metres (m)
risk: severity combined with likelihoodUnit: ordinal score

At v=1 m/s and reaction t=0.2 s, d_react=1×0.2=0.2 m.
With braking deceleration a=2 m/s², d_brake=1²/(2×2)=0.25 m.
Ideal total is 0.45 m; add measured margin for sensing, actuation, floor, load, and uncertainty.

Programmer analogy

A security threat model maps assets and controls; robot safety adds kinetic energy and independently enforced stops.

At 0.5 m/s with 0.2 s reaction time, what is reaction distance?

0.5×0.2 = 0.1 m.

For $I=3$ interventions in $N=30$ trials,

r_{\mathrm{intervention}}=\frac{I}{N}=\frac{3}{30}=10\%.

Report autonomous success, intervention-assisted completion, safe aborts, and unsafe events as separate outcome counts.

Reconcile a 30-trial outcome ledger

A frozen suite yields 21 autonomous successes, 3 assisted completions, 2 partial completions, 3 safe aborts, and 1 unsafe physical outcome. Five trials contain actual human intervention, and 7 of 300 action proposals are rejected for violating a defined constraint.

Verify the terminal buckets sum to the denominator: 21 + 3 + 2 + 3 + 1 = 30; investigate any missing or double-counted trial before computing rates.
Compute autonomous success as 21/30 = 70% and assisted completion as 3/30 = 10%, keeping them separate rather than reporting 80% autonomous.
Compute partial completion 2/30 = 6.7%, safe abort 3/30 = 10%, and unsafe physical outcome 1/30 = 3.3% with raw counts beside rounded percentages.
Compute trial intervention rate as 5/30 = 16.7%; explain why five can exceed the three assisted completions because intervention may occur in a later partial, abort, or unsafe trial.
Compute guard-rejection rate over proposals as 7/300 = 2.33%; separately record how many rejected proposals were unsafe, how many were false positives, and whether any prohibited effect escaped.
Slice the ledger by nominal, ambiguity, stale-state, injection, and blocked-path families; report the worst family and link every count to immutable trace IDs.

Result

The scorecard exposes 70% autonomous success, 16.7% intervention, 10% safe abort, 3.3% unsafe physical outcome, and 2.33% proposal rejection without hiding assistance or danger inside completion.

What this proves

A metric is credible only when its category, denominator, overlap rules, raw count, and trace lineage are visible.

Physical examples

Where this appears in real life

Operator points to the intended tote

The robot asks which of two identical totes is intended; the operator points, and the robot completes the delivery.

Look for:

The terminal category is assisted completion, the clarification and intervention counters increment, and the trial is excluded from autonomous success without calling it a failure of containment.

Guard blocks a fast approach

A policy proposes excessive speed, the validator rejects it, and the planner selects a slower valid skill that completes autonomously.

Look for:

The trial may end as autonomous success under the declared assistance rule, while unsafe-proposal and guard-rejection counters expose the upstream defect and downstream containment.

Hands-on exercise

Make the idea observable

Use ten synthetic or previously recorded traces. Do not rerun a failed case to replace its label, and do not create a single blended safety-and-success score.

Write terminal definitions and cross-cutting event definitions on one page, including allowed observation-only operator behavior and what counts as intervention.
Label each trace once as autonomous success, assisted completion, partial, safe abort, or unsafe physical outcome; record evidence and reviewer confidence.
Count clarification, actual intervention, retry, recovery attempt, unsafe proposal, guard rejection, controller rejection, deadline miss, and forbidden event separately.
Calculate rates with explicit denominators, then verify every terminal count sums to ten and every event count links to a trace and timestamp.
Have a second reviewer independently label three difficult traces, compare disagreements, and refine definitions without changing the original raw evidence.
Build a compact table showing totals, scenario slices, worst slice, missing evidence, and one sentence describing what the small suite cannot support.

Observe

Most disagreements arise at the boundary between clarification and intervention, safe abort and partial completion, or rejected unsafe proposal and unsafe physical outcome.

Done when

Terminal counts reconcile exactly, event denominators are appropriate, ambiguous labels are documented, every number links to evidence, and no aggregate hides an unsafe or assisted outcome.

Build today

Create a scenario runner that measures task success, intervention, recovery, latency, and unsafe attempts.

Evidence to save

DONE when a comparison table for “Success, partial success, intervention, and safety metrics” contains the test condition, metric, result, and justified engineering decision.

Common mistakes

Catch the wrong mental model

Wrong

Adding autonomous and assisted completion and calling the result autonomy.

Better

Report task completion and autonomous success separately; any human action outside the predeclared allowance remains visible as intervention.

Wrong

Counting a blocked unsafe proposal as if no safety-relevant event occurred.

Better

Credit containment while also counting the unsafe proposal and guard action; downstream success does not erase upstream risk.

Wrong

Using trials as the denominator for every metric.

Better

Match the denominator to the question: trials for terminal outcomes, proposals for proposal defects, eligible failures for recovery, and defined requests for latency.

Job connection

How this becomes employable evidence

Define and automate an embodied-agent scorecard that reconciles trial outcomes, interventions, guard activity, recovery, latency, and unsafe behavior across scenario slices while preserving raw traces for release review.

Relevant target roles

Robotics Deployment, Integration & Validation Engineer
Robot Learning Deployment / Physical AI Integration Engineer
Robot Fleet Backend / Platform Engineer
Robot HMI / Control & Monitoring Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 21 interview drill

Interview questions: Success, partial success, intervention, and safety metrics

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A team reports 80% success from 21 autonomous and 3 assisted completions in 30 trials. Correct the claim, design the remaining outcome and event metrics, and explain how guard rejections can indicate both containment and poor policy quality.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why do the five interventions not have to equal the three assisted completions?

Model interview answer

Intervention can occur in trials that later end partial, abort, or unsafe; assisted completion is only one mutually exclusive terminal category.

Q2What is the correct autonomous-success rate in the worked ledger?

Model interview answer

It is 21 out of 30, or 70%; the three assisted completions must not be added to autonomy.

Q3What must accompany a zero-unsafe-event result?

Model interview answer

Scenario coverage, denominator, definitions, build and environment versions, evidence completeness, guard activity, limitations, and a scoped claim rather than a universal safety statement.