Reading digits: pixels, convolutions and softmax
How a small network reads a handwritten digit: a weight for every pixel, local patches, shared filters (convolution), pooling and a softmax vote, with a live drawing pad.
This is the text of a chapter of the animated talk A Deep Seek into LLM Architecture by Ruben Galvão . Open the page with scripts on to play the slides, one key press per idea.
A picture as input A picture is a grid of numbers: 28×28 = 784 pixels, 0 to 1 That is what we feed the network One neuron, every pixel One neuron sees every pixel: 784 weights for ONE neuron FLOPs now per image: 784 multiplies + 784 adds Small models count every op; from AlexNet on, only multiplies/adds (the rest < 1 %) A column of neurons A column of 26 neurons, each seeing every pixel: ≈ 20 thousand weights The layer as an image Unfold to 26×26 = 676 neurons, outputs as brightness Random weights → noise; over half a million weights Treats the picture as a long list: no idea pixels are neighbours Over 1 million operations per image Local patches Idea one: each neuron looks only at its 3×3 patch 9 weights instead of 784 Output follows the digit’s shape (weights still random) Shared weights: a filter Idea two: share the same 9 weights everywhere One filter slid over the picture: convolution The map lights up where one edge appears Same work (12,168 ops), only 10 weights Eight filters Eight filters, each its own pattern → a block, 26×26×8 Each output through a ReLU (counted) Lines: one cell looks at a small patch of the previous layer Max-pooling, and again Max-pool shrinks each map; next filters are 3×3×8, sixteen of them Pool again: 5×5×16; flatten to 400 numbers Conv, pool, conv, pool: AlexNet’s shape, just bigger ≈ 389 thousand ops per image so far The vote: ten neurons Vote: ten neurons, one per digit, each over all 400 numbers Raw scores: positive = “me”, negative = “not me”; any size Counter: 396,728 ops Softmax Softmax: e^score, divide by the total → probabilities summing to 1 99 % sure it’s a 7; same trick picks the next word later 29 more ops → 396,757 to read one digit Live: draw a digit Live: draw a digit, watch it flow through Try something ugly or a letter: still confident; it only knows what it was shown Papers and sources LeCun, Cortes & Burges (1998): The MNIST database of handwritten digits LeCun et al. (1998): Gradient-based learning applied to document recognition (LeNet-5) John Bridle (1990): Probabilistic interpretation of feedforward classification network outputs (softmax)