Matrix multiplication as transformation

1

Concept

Matrix multiplication is easiest to understand as a transformation, not a grid-filling ritual. A matrix $W$ takes coordinates in one space and produces coordinates in another. For an input vector $x$, each output component of $Wx$ is a dot product between one row of $W$ and $x$. The rows therefore ask different learned weighted questions about the same input.

Consider

Applied to $(x_1,x_2)$, it produces $(2x_1,x_2/2)$. A square drawn in the plane becomes a rectangle: the horizontal direction stretches while the vertical direction shrinks. Other matrices rotate, reflect, shear, or project. In higher-dimensional neural networks, the pictures disappear but the operation remains a learned change of representation.

A rectangular matrix can also change dimensionality. An $m\times n$ matrix maps $n$ input features to $m$ output features. If $m<n$, it may compress information; if $m>n$, it embeds the input in a larger coordinate space but does not magically create independent information. The matrix’s rank measures how many independent directions survive. A low-rank map forces outputs into a smaller subspace, an idea later exploited by parameter-efficient fine-tuning.

Matrix-matrix multiplication performs many transformations or examples together. Entry $(i,j)$ of $AB$ is the dot product of row $i$ of $A$ with column $j$ of $B$. More importantly, $AB$ represents composition: $B$ acts first on a vector, then $A$. Matrix multiplication is generally not commutative. $AB$ and $BA$ can differ, and one may be defined when the other is not. Shape records which compositions make sense.

Neural layers commonly use $y=Wx+b$. The matrix supplies a linear transformation; the bias vector translates every result. This is technically an affine transformation because zero can map to $b$ rather than zero. Bias lets a decision boundary move away from the origin. Implementations may store examples as rows and write $XW+b$, but this is only a convention. Clear shape annotations prevent transpose confusion.

If we stack two linear layers without anything between them, $W_2(W_1x)=(W_2W_1)x$. The stack is equivalent to one matrix. Biases also combine into one affine map. Depth becomes expressive only when nonlinear functions, gating, normalization, or input-dependent routing interrupt that collapse. This is why activation functions appear between learned projections.

Matrix multiplication dominates neural computation because hardware can execute dense blocks efficiently. A batch of token vectors multiplied by one weight matrix reuses the same parameters across positions. GPUs and specialized accelerators divide large products into tiles, move them through memory hierarchies, and accumulate partial sums. The algebra is simple; performance often depends on data movement, shapes, precision, and how well operations are fused.

Transformers repeatedly apply this pattern. Embedding matrices map token identities to features. Attention projections create queries, keys, and values. Output projections recombine heads. Feed-forward layers expand and contract feature dimensions. The language-model head maps hidden states to vocabulary logits. Thinking in transformations makes the architecture legible: at every matrix, ask which space enters, which space leaves, what directions may be preserved, and what information could be lost.