Chapter 17 · Prove FleetOps across regression, interfaces, safety, and commissioning
Today in the field story
One problem, then the next
Rehearse FAT in the pinned environment, SAT against the declared site assumptions, and commissioning with an immutable checklist. When R-17 fails, separate the observed timeline from hypotheses, identify the earliest supported cause, apply a corrective change, rerun affected and regression cases, and prove rollback. Issue the go or no-go record with residual risks. Preserve this package because Week 25 will turn the same gates into continuous operations.
- Why now
The spiral closes only when test evidence produces a customer-facing decision and a recoverable incident record.
- Ignore today
Ignore schedule pressure and demo polish; make every deviation and blocker explicit.
- Unlocks next
A validated FleetOps baseline for embodied-AI integration and later production operations.
Understand
Build the physical picture first
Commissioning is a controlled transfer of evidence and responsibility: factory proof, site-specific proof, unresolved deviations, trained ownership, rollback, and support readiness must agree before production use.
Factory Acceptance Testing exercises the integrated system against agreed requirements before delivery or deployment in the supplier-controlled environment. Site Acceptance Testing repeats the relevant acceptance evidence with the customer’s utilities, network, maps, interfaces, environmental conditions, payloads, workflows, safeguards, and operators. A pleasant factory demo or a successful site smoke mission is not FAT or SAT unless the frozen matrix, witnesses, configurations, measurements, deviations, and approval rules are present.
Commissioning includes installation verification, power and network checks, coordinate frames and maps, calibrations, payload and speed configurations, WMS/WES and PLC handshakes, fleet capacity, charging, user roles, safety controls, operator and maintenance training, backups, recovery, rollback, spares, monitoring, escalation, and ownership handoff. Site acceptance should occur before initial production startup, and changed conditions later require impact review and appropriate revalidation rather than assuming the original signature lasts forever.
Acceptance is requirement-based, not a percentage contest. Fifty passing convenience cases cannot cancel one failed protective function, data-loss control, or mission-critical rollback. Classify blockers, deviations, conditional acceptance, and residual risks before the run; require named authority for each disposition. A SAT pass after a fix does not erase the original FAT failure—keep the defect, containment, causal evidence, corrective change, and regression link intact.
An incident record separates observation from inference. Preserve timeline, versions, configuration, commands, logs, traces, bags, site conditions, impact, containment, and recovery. Call a cause “root cause” only when evidence explains the failure and corrective action prevents the relevant recurrence. If evidence is missing, state competing hypotheses and keep the affected requirement blocked. The FleetOps gate closes only when required evidence, safe-failure tests, rollback drill, training, handoff, and go/no-go authority are complete.
Words you need
Name each idea precisely
- Factory Acceptance Test
A witnessed, requirement-linked evaluation of the integrated system in the supplier or factory-controlled environment before shipment or site release.
Physical example:The frozen FleetOps bundle runs nominal, degraded, rollback, and interface cases against the agreed simulated and bench matrix with signed deviations.
- Site Acceptance Test
A witnessed evaluation that the installed system satisfies agreed requirements with the site’s actual utilities, interfaces, environment, workflows, and operating constraints.
Physical example:The customer site checks Wi-Fi coverage, mapped aisles, charger access, conveyor timing, payloads, operator roles, safeguards, and recovery using the approved matrix.
- Commissioning
The controlled process of verifying installation, configuration, interfaces, safety and operational readiness, training, documentation, support, and responsibility transfer.
Physical example:The release pack includes maps, calibration, backups, rollback, spare and escalation paths, operator training, maintenance ownership, and a signed production-start decision.
- Deviation
A documented difference from an agreed requirement or procedure with impact, containment, owner, expiry, and authorized disposition.
Physical example:A non-critical dashboard label is accepted temporarily with a dated correction, while a failed protective response remains an unconditional blocker.
- Corrective-action verification
Evidence that a change addresses the supported cause and that focused plus broader regression does not introduce unacceptable behavior.
Physical example:A stale-state race fix passes the original failure replay, adjacent reconnect cases, full mission regression, and the signed SAT scenario.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalRisk priority, trial denominators, and recovery time
Validation turns hazards into traceable tests and reports every planned trial, including the failures that make a result uncomfortable.
- RPN = S×O×D
- an ordinal FMEA priority from severity, occurrence, and detection ratingsUnit: relative score
- p̂ = k/N
- observed passes k divided by all planned trials NUnit: fraction or percent
- Tᵣ
- time from a declared failure event until every recovery condition remains trueUnit: seconds (s)
A hazard is rated severity S=5, occurrence O=2, and detection difficulty D=4, so its relative RPN is 5×2×4 = 40.
A frozen matrix planned N=20 trials and passed k=17, so the observed pass rate is 17/20 = 85%; the three failures stay in the denominator.
If valid sensing returns at 12.0 s and all stability criteria hold from 15.5 s onward, report recovery time Tᵣ = 3.5 s and preserve the trace.
Treat it like a release test matrix with trace IDs, except the failed requirement can concern motion, collision, or loss of control rather than a screen defect.
A frozen suite passes 27 of 30 planned trials. What observed pass rate must be reported?
27/30 = 0.90 = 90%, with all three failures retained and categorized.
Factory acceptance is
with failures. Site acceptance is
but it does not erase the earlier failure records.
Decide a gate with high pass percentage and two critical failures
The FAT matrix contains 55 required cases. Fifty-three pass, one safety-related reset case fails, and one rollback case corrupts mission configuration. After correction, the site later records 55 of 55 SAT cases passing.
Calculate the initial FAT percentage as
53 / 55 × 100 ≈ 96.4%, then keep the two failures visible by requirement and criticality rather than rounding the release to success.Apply the predeclared gate: the reset and rollback cases are blocking, so shipment or production release is no-go despite the 96.4% total.
Contain the issues, preserve both failure bundles, document observed facts, investigate supported causes, and link each corrective change to its original requirement, risk, and result.
Rerun each exact failure case plus adjacent reset, configuration migration, canary, rollback, and complete-mission regression under the new immutable bundle.
At site, execute all 55 SAT cases with actual network, maps, payload, equipment emulators or approved interfaces, roles, training, backup, recovery, and witness records.
Record the final go decision only after 55 of 55 site cases and every non-test commissioning item pass; retain the original FAT failures and corrective history in the signed package.
The system moves from no-go at 96.4% FAT to an evidence-supported site decision only after both critical failures are corrected, regressed, and preserved in history.
Acceptance authority follows critical requirements and complete commissioning evidence, never the comfort of a high aggregate pass rate.
Physical examples
Where this appears in real life
Theater rehearsal and opening night
A cast rehearses the complete show on a practice stage, then checks the destination venue’s lighting, exits, rigging, cues, staff roles, and emergency procedures before admitting an audience.
Factory rehearsal proves integration under controlled conditions; the venue still introduces interfaces and responsibilities that require witnessed site evidence.
House key and maintenance handoff
A paper handoff pack contains inspection results, open defects, utility shutoffs, warranty contacts, spare keys, maintenance schedule, emergency numbers, and signatures.
Possession of the key is not operational readiness; evidence, unresolved items, knowledge, recovery, and ownership must transfer together.
Hands-on exercise
Make the idea observable
Run a simulated commissioning rehearsal for the FleetOps lab. Label all physical-site, qualified safety, and unavailable HIL steps as witnessed placeholders or blockers rather than pretending they ran.
Generate FAT and SAT matrices directly from the traceability table, naming environment-specific cases, witnesses, immutable bundle identities, required artifacts, blocker classes, and deviation authority.
Run the available FAT suite, including nominal missions, duplicate requests, stale state, adapter restart, blocked route, equipment timeout, safe cancellation, canary rejection, and complete-bundle rollback.
Create a site-delta checklist for Wi-Fi and time service, maps and frames, floor and payload, chargers, WMS/WES and PLC interfaces, user roles, safeguards, training, backups, monitoring, spares, escalation, and support ownership.
Inject one site-only failure, preserve facts and impact, contain it, write competing hypotheses, prove a supported cause where possible, implement one corrective action, and rerun focused plus broad regression.
Rehearse rollback from the candidate application, configuration, schema, calibration, and interface bundle to the known-good bundle; verify terminal mission consistency and audit history after recovery.
Publish the signed-style gate record with passed, failed, blocked, and not-applicable counts; deviations; unresolved risks; training and handoff status; artifact links; and explicit go, conditional-go, or no-go authority.
The site-delta review exposes dependencies absent from the factory, and the incident rehearsal tests whether the team can preserve evidence and recover without rewriting history.
Every available requirement has witnessed evidence, unavailable physical and safety obligations remain visibly blocked, rollback restores a coherent bundle, and the gate decision follows predeclared authority.
Build today
Create a risk-linked SIL→HIL acceptance ladder for FleetOps, automate regression scenarios, integrate one external fleet or PLC boundary, and publish FAT/SAT evidence plus an incident report.
Evidence to save
DONE when the weekly ship note explains how “FAT, SAT, customer commissioning, incident reporting, and FleetOps gate” changed the build, what still fails, and the first task for next week.
Project gate
Gate 4 · FleetOps passes its risk-linked SIL/HIL acceptance matrix, safe failure tests, rollback drill, and commissioning review.
Common mistakes
Catch the wrong mental model
Approving release from aggregate pass percentage while a critical requirement fails.
Apply predeclared requirement criticality and blocker rules; preserve failed evidence and require authorized corrective-action verification before reconsidering the gate.
Treating SAT as a repeat of the easiest factory smoke test.
Exercise the customer’s actual utilities, network, environment, maps, payloads, interfaces, safeguards, workflows, roles, recovery, and witness requirements.
Naming the first plausible explanation as root cause.
Separate facts from hypotheses, preserve missing evidence, reproduce where possible, and promote a cause only when the corrective action explains and prevents the relevant recurrence.
Rolling back only the application while leaving incompatible configuration or schema changes.
Version and rehearse rollback for the complete application, configuration, schema, calibration, firmware, and interface bundle, including data and mission reconciliation.
Job connection
How this becomes employable evidence
Own a customer commissioning package that ties factory and site matrices to requirements, coordinates robot, fleet, network, WMS/WES, PLC, safety, operator, support, and rollback owners, and keeps incidents and deviations evidence-complete.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robot Fleet Backend / Platform Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 17 interview drill
Interview questions: FAT, SAT, customer commissioning, incident reporting, and FleetOps gate
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
FAT passes 53 of 55 cases and SAT later passes everything. Explain why release may still be blocked, what site acceptance adds, how you preserve the two failures, and what must be handed over before production starts.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What does SAT add after a successful FAT?
It verifies the installed system against agreed requirements with the site’s real utilities, networks, environment, interfaces, payloads, workflows, safeguards, operators, and witnesses.
Q2Should a later SAT pass remove an earlier FAT failure record?
No. Keep the original failure, containment, cause evidence, corrective change, focused and broad regression, and final disposition linked.
Q3When should an incident report say the root cause is unknown?
When available evidence does not uniquely explain the failure or a corrective action has not demonstrated prevention; list facts, containment, hypotheses, and missing evidence instead.
Chapter starter artifact
Issue a release decision from complete evidence
A bidirectional acceptance package traces R-17 and the remaining FleetOps risks to component, launch, deterministic simulation, SIL, HIL, industrial-interface, safety-awareness, FAT, SAT, incident, corrective-action, and rollback evidence, ending in an explicit reproducible go/no-go decision.
const cases = [
{ id: "R17-unit", status: "pass", evidence: true },
{ id: "R17-launch", status: "pass", evidence: true },
{ id: "R17-stale", status: "fail", evidence: true },
{ id: "R22-dock", status: "pass", evidence: false },
{ id: "R30-schema", status: "unknown", evidence: "false" },
];
const validStatus = new Set(["pass", "fail"]);
const schemaValid = (test) =>
typeof test.id === "string" && test.id.length > 0 &&
validStatus.has(test.status) && typeof test.evidence === "boolean";
const validCases = cases.filter(schemaValid);
const invalidSchema = cases.length - validCases.length;
const passed = validCases.filter(
(test) => test.status === "pass" && test.evidence,
).length;
const failed = validCases.filter((test) => test.status === "fail").length;
const missingEvidence = validCases.filter((test) => test.evidence === false).length;
const decision =
failed === 0 && missingEvidence === 0 && invalidSchema === 0
? "PASS" : "NEEDS REVISION";
const output =
"decision=" + decision +
" passed=" + passed +
" failed=" + failed +
" missingEvidence=" + missingEvidence +
" invalidSchema=" + invalidSchema;
const expected =
"decision=NEEDS REVISION passed=2 failed=1 missingEvidence=1 invalidSchema=1";
if (output !== expected) throw new Error("release gate mismatch: " + output);
console.log(output);Download the file into your terminal's current folder, then run the command below. The expected output is exact.
- Run
node week-17-fleetops-release-gate.mjs
- Expected output
decision=NEEDS REVISION passed=2 failed=1 missingEvidence=1 invalidSchema=1
- Planted failure to diagnose
The dock case lacks evidence, the stale-authority case fails, and the final row has an unknown status plus string evidence; truthiness or ignored statuses would issue a false release.