Vectors, matrices, and dot products

1

Concept

A vector is an ordered list of numbers. Order matters: $(2,5)$ is not the same vector as $(5,2)$. In geometry, a two- or three-dimensional vector can represent a direction and length. In machine learning, a vector may have hundreds or thousands of coordinates and represent a token, an image patch, a hidden state, or a gradient. We cannot draw all those dimensions, but the same arithmetic still applies.

Vectors can be added component by component and multiplied by a scalar. If $a=(1,2)$ and $b=(3,-1)$, then $a+b=(4,1)$. Multiplying $a$ by three gives $(3,6)$. The Euclidean length, or norm, is $\lVert a\rVert=\sqrt{1^2+2^2}$. Length often expresses magnitude, while direction captures the relative pattern across coordinates. Neural networks continually reshape both.

The dot product combines two equal-length vectors into one scalar:

For the vectors above, $a\cdot b=1\times3+2\times(-1)=1$. Each matching pair contributes positive evidence when signs agree and negative evidence when they oppose. Geometrically, $a\cdot b=\lVert a\rVert\lVert b\rVert\cos\theta$. Vectors pointing in similar directions have a positive dot product; perpendicular vectors have zero; opposing directions produce a negative value.

This interpretation needs one caution. A raw dot product mixes direction with length. Two vectors can score highly because they align, because their norms are large, or both. Cosine similarity divides by the norms and isolates directional similarity: $(a\cdot b)/(\lVert a\rVert\lVert b\rVert)$. Embedding applications sometimes use cosine similarity, while attention deliberately uses scaled dot products whose magnitudes remain part of the learned computation.

A matrix is a rectangular array of numbers. Its shape is written rows by columns. A $3\times4$ matrix can map a four-component input vector to a three-component output. Each output coordinate is the dot product of one matrix row with the input. This is why matrix-vector multiplication is central to neural networks: it computes many learned weighted sums at once.

Shapes are a practical type system. Multiplying an $m\times n$ matrix by an $n$-component vector yields an $m$-component vector. The inner dimensions must agree. For batched data, rows often hold examples or token positions, so a matrix $X$ with shape sequence-by-features multiplies a weight matrix with shape features-by-output. Writing shapes beside equations catches many bugs before code runs.

Neural features are not usually human-named sliders. A coordinate may participate in many patterns, and a concept may be distributed across many coordinates. Rotating the coordinate system can preserve relationships while changing individual values. Consequently, inspecting one dimension rarely reveals a clean semantic label. Geometry is most useful through relationships: distances, directions, subspaces, projections, and how learned transformations change them.

The bridge to language models is direct. An embedding lookup returns a vector for each token ID. Projection matrices turn hidden states into queries, keys, values, or vocabulary logits. Query-key dot products create attention scores. Gradients are vectors pointing toward local change in loss. Once vectors are seen as states, matrices as learned maps, and dot products as weighted alignment, much of the Transformer becomes repeated linear algebra with carefully placed nonlinear operations.