Lesson 5 of 6 · Batches and tensor shapes

How can one model handle eight requests at once?

You do not need eight copies of a model to process eight requests. Put the inputs in separate rows of a matrix and apply the same weights to every row. This is batching.

Step 1 · Add an examples dimension

A batch is many inputs, not many copies of the model

Let B be the number of requests and F the features per request. Stacking the feature rows gives X[B,F]. Every row uses the same weights and bias. In the example below, we expand the router to three input features and five output classes.

During training, averaging across suitably sampled examples can give a less noisy gradient than using one example. During inference, batching lets the accelerator perform several request calculations together. The math shape is similar; backward and parameter updates happen only in training.

Four tickets, one loaded router

X [4, 3]Four isolated ticket inputs, each represented by three features.
W [3, 5]One shared parameter matrix mapping three features to five classes.
Z [4, 5]Five output logits for each of the four tickets.

Quick check

If X is [8,3] and W is [3,5], what is X @ W?

[3,5][8,5][8,3]

Step 2 · Follow the rows and columns

The feature dimensions must match

Let C be the number of output classes. In X @ W, each request’s F features multiply F weights for each class. The bias has one value per class and is added to every request row.

Shape trace

X[B,F] @ W[F,C] + b[C] → Z[B,C]

For one output cell, Z[request, class] is the dot product of that request’s feature row with that class’s weight column, plus the class bias.

Practical example · Shape debugger

Change each dimension and predict what survives

Increase batch size and confirm that weights do not grow. Increase class count and notice that both the weight columns and output columns do.

Batch size, B Input features, F Output classes, C

Examples

Weights

Bias

Logits

The result contains logit values: one for every request/class pair.

Step 3 · Define reduction and gradient state

Three batch bugs have three different symptoms

Batching also changes how training combines losses and gradients. Check these three mistakes separately. Keeping gradients across batches is valid when you deliberately use gradient accumulation; it is a bug when the next update was meant to use only the current batch.

Bug A · Sum where the config assumes mean

What changes silently when batch size doubles?

The number of output classesThe effective update sizeThe number of input features

Bug B · Average twice

What happens when loss is averaged and the already-averaged gradient is divided by B again?

Gradients become too largeGradients become too smallNothing changes

Bug C · Keep stale gradient buffers

What does the next optimizer step represent?

Only the current batchThe current batch mixed with old accumulated gradientsAn inference-only forward pass

Python bridge · vectorized shared parameters

logits = X @ W + bias        # [B,F] @ [F,C] + [C] → [B,C]
loss = cross_entropy(logits, labels, reduction="mean")
loss.backward()              # training only

Inference serving uses the first line too, often with a much larger hidden dimension and vocabulary. It simply omits loss, backward, and optimizer mutation.

Your turn

Explain it in your own words.

Not checked

Eight requests each supply three features, so X has shape [8,3]. One model has weights W shaped [3,5]. In X @ W, what does each output row represent, how many scores does it contain, and why don't we need eight copies of W?

Answer the question in your own words. A short explanation is enough.

Your explanation

Draft saves on this device0/800 characters

Work through a hint