Chapter 13 · Build a trustworthy PyTorch experiment from robot-shaped data

Today in the field story

One problem, then the next

Establish a simple numeric or constant baseline before introducing a learned encoder. Compare an MLP on measured features with a compact convolutional path on image structure, tracing receptive field, parameters, tensor shapes, and output units. The mission does not reward the fashionable model; it rewards the smallest model that improves the declared seal-offset task without hiding latency or input assumptions.

Why now: Architecture choices only have meaning relative to input structure, baseline, and output contract.
Ignore today: Ignore pretrained giants and exhaustive search; compare two understandable candidates.
Unlocks next: A selected scout architecture whose behavior can be trained and measured.

Understand

Build the physical picture first

An MLP is a team where everyone reads every input, while a convolutional encoder reuses one small detective across an image to find local patterns.

A neural network is a parameterized function built from modules. A linear layer computes weighted sums plus biases; without a nonlinear activation, stacking linear layers still collapses into one linear transformation. ReLU keeps positive values and maps negative values to zero, allowing the network to form piecewise nonlinear decisions. An MLP is suitable when a fixed-size observation vector has meaningful features such as joint positions, velocities, ranges, and task state. The output layer must match the physical target contract exactly, whether that is two pose coordinates, class logits, or a bounded action representation.

A dense layer connects every input to every output. Its parameter count is input_features × output_features + output_features because each output has one weight per input and one bias. Flattening a 64-by-64 RGB image into 12,288 inputs and connecting it to 128 hidden units requires more than 1.5 million parameters before the next layer. That may memorize small data, ignore image locality, and exceed an edge latency or memory budget. Parameter count is a capacity and deployment clue, not a direct score of intelligence or quality.

A convolution slides the same small kernel across spatial positions. Weight sharing lets one edge or corner detector operate on the left, middle, or right of an image. The input and output still have batch and channel axes, and stride, padding, dilation, and kernel size determine spatial shape. Deeper layers combine local patterns into a larger receptive field. This translation-related bias often makes a CNN a better visual encoder than a fully connected image layer, but it does not make the model invariant to lighting, scale, viewpoint, occlusion, or camera changes.

Architecture selection starts with the task contract and a baseline. Trace every intermediate shape, calculate parameters, inspect activation ranges, and time inference on the intended class of device. Compare a constant or simple linear baseline before adding depth. A smaller model that meets held-out error and latency may be the correct robot component. A model output is still only a prediction: downstream software must validate freshness, finite values, units, frames, confidence or quality rules, and action limits before it can affect any simulated or physical execution.

Words you need

Name each idea precisely

Linear layer: A module that maps an input vector to weighted sums plus one bias per output feature.
Physical example:
Ten normalized robot-state values feed eight hidden values through an nn.Linear(10, 8) layer containing 88 parameters.
Activation function: A nonlinear operation between parameterized layers that prevents the whole network from reducing to one linear map.
Physical example:
ReLU passes a positive feature response and replaces a negative response with zero before the next layer.
MLP: A multilayer perceptron made mainly from dense linear layers and nonlinear activations for fixed-size feature vectors.
Physical example:
Joint angles, velocities, gripper state, and target offset form one vector used to predict a two-number correction.
Convolution: A local weighted operation whose kernel parameters are reused across spatial positions.
Physical example:
The same three-by-three filter responds to a vertical cup edge near either side of the camera image.
Channel: A feature-map axis; input channels may be colors and later channels represent learned pattern responses.
Physical example:
A convolution turns three RGB channels into sixteen learned maps while preserving the declared image height and width.
Receptive field: The region of the original input that can influence one later feature value.
Physical example:
One early feature sees a three-by-three patch, while stacked convolutions let a later feature combine evidence from a wider cup region.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalTensor shapes and gradient descent

Most robot-learning bugs are shape, scale, split, or optimization mistakes.

B×T×D: batch, time steps, and feature dimensionsUnit: counts
L: loss measuring prediction errorUnit: task-dependent
η: learning rateUnit: unitless scale

A batch has B=8 episodes, T=20 time steps, and D=12 features.
The tensor contains 8×20×12 = 1,920 numbers.
An optimizer updates a weight with w_new = w_old − η∂L/∂w; inspect shape and finite values before training.

Programmer analogy

Tensor shapes are explicit multidimensional array contracts: every axis has a declared meaning and size.

How many values are in a 4×10×6 tensor?

4×10×6 = 240 values.

A dense layer has

(d_{\mathrm{in}}+1)d_{\mathrm{out}}=(10+1)(8)=88\ \text{parameters}.

A $3\times3$ convolution from $3$ to $16$ channels has $(3\times3\times3+1)\times16=448$ parameters. MSE may train numeric outputs, but it does not choose the architecture.

Compare one dense layer with one image convolution

A robot-state MLP maps 10 input features to 8 hidden features. A visual encoder applies a 3-by-3 convolution from 3 RGB channels to 16 output channels, with one bias per output channel, stride 1, and padding 1 on a 32-by-32 image.

Count dense weights: 10 × 8 = 80, because every one of eight outputs reads all ten input features.
Add eight dense biases: 80 + 8 = 88 parameters for Linear(10, 8).
Count one convolutional filter: 3 × 3 × 3 = 27 weights because it spans height, width, and all three input channels.
Multiply by 16 output channels and add 16 biases: (27 + 1) × 16 = 448 convolution parameters.
Use output-size arithmetic: (32 + 2×1 - 3) / 1 + 1 = 32, so output shape is [B, 16, 32, 32].
State the decision boundary: these counts explain structure and cost, but held-out error, failure slices, and measured inference latency decide whether either encoder serves the task.

Result

The state layer has 88 parameters; the image convolution has 448 shared parameters and preserves 32-by-32 spatial size while creating 16 learned channels.

What this proves

Calculate connectivity and shapes before training, then choose the smallest architecture that earns its complexity against a baseline under robot-relevant tests.

Physical examples

Where this appears in real life

Everyone reads every gauge

Give ten gauge cards to each of eight pretend technicians and let every technician choose a different weight for every gauge before reporting one value.

Look for:

The full set needs eighty connection weights plus eight biases, matching a dense 10-to-8 layer and showing how connections grow.

Sliding stencil detector

Cut a three-by-three stencil, assign a small score to each opening, and slide the same stencil over a drawn grid containing the same corner in several places.

Look for:

One shared scoring pattern detects the corner at multiple locations instead of learning a separate detector for every pixel position.

Hands-on exercise

Make the idea observable

Use CPU-only PyTorch and synthetic feature vectors or generated geometric images. Keep the output disconnected from any live controller.

Write an explicit input and output contract for either a ten-feature observation-to-two-value model or a small image-to-two-coordinate model.
Implement a linear baseline, a tiny MLP, and—only for image input—a tiny CNN using named nn.Module parts with an inspectable forward path.
Feed one nominal batch and one batch with a different batch size; assert every intermediate shape and the exact final target shape.
Calculate parameter counts by hand for one dense and one convolutional layer, then compare them with PyTorch’s trainable-parameter count.
Inspect output and activation minima, maxima, means, and finite status; deliberately pass the wrong channel order and make the contract fail before inference.
Time repeated evaluation-mode, no-gradient CPU calls after warm-up, save architecture, shapes, parameter table, device, timing method, and a baseline comparison plan.

Observe

The MLP removes the feature axis into hidden values, the CNN preserves spatial organization until deliberate pooling or flattening, and parameter counts match the hand calculation.

Done when

A reviewer reproduces the 88 and 448 counts, follows every tensor shape, sees the wrong-layout assertion fire, and understands why no model-quality claim was made from architecture alone.

Build today

Train a small image-to-pose or observation-to-action network with a reproducible experiment report.

Evidence to save

DONE when a deterministic “MLPs and convolutional visual encoders” failure test reports expected versus actual behavior and passes after the documented fix.

Common mistakes

Catch the wrong mental model

Wrong

Adding more dense layers without nonlinear activations and expecting a more expressive nonlinear model.

Better

Place an appropriate nonlinear activation between linear layers and inspect the resulting activation and output ranges.

Wrong

Flattening a large image immediately because the code becomes shorter.

Better

Preserve spatial structure with a suitable visual encoder, calculate the dense parameter explosion, and compare latency and held-out performance.

Wrong

Choosing the architecture with the most parameters before defining a baseline or deployment budget.

Better

Start with the simplest credible baseline and require added capacity to improve frozen evaluation slices within measured memory and latency limits.

Job connection

How this becomes employable evidence

Select an observation encoder for a robot-learning component by tracing input semantics and shapes, calculating parameter and memory cost, measuring edge inference latency, and defining the validation and runtime guards around its output.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 13 interview drill

Interview questions: MLPs and convolutional visual encoders

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Compare an MLP and CNN for predicting a camera-relative target. Derive one parameter count, explain receptive field and weight sharing, and name evidence beyond training loss that decides the design.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why does stacking only linear layers still describe one linear transformation?

Model interview answer

Composing linear or affine maps can be algebraically combined into another affine map; nonlinear activations create boundaries that one affine map cannot represent.

Q2Why can a convolution use fewer parameters than a full image-sized dense layer?

Model interview answer

It reuses the same small local kernel across positions rather than learning a separate weight from every pixel to every output.

Q3What does a correct parameter count fail to prove?

Model interview answer

It does not prove suitable data, low held-out error, robustness, runtime latency, calibrated outputs, or safe robot integration.