Orrery

Computing

How does a machine learn from examples?

Nobody writes the rules into a neural network. It starts as a pile of random numbers, makes guesses, measures how wrong they are, and nudges every number a little in whichever direction makes it less wrong. Here you can do each of those steps yourself.

A common picture of artificial intelligence is a program full of rules that someone typed in: if the email mentions a prize, it is spam. Neural networks work the other way round. Their makers write down a formula with many adjustable numbers in it, show it thousands of examples (inputs together with the right answer), and let a simple procedure tune the numbers until the formula gives the right answers. Nobody chooses the final numbers, and nobody writes a rule.

To see how that can work, start with the smallest possible piece.

One neuron draws a line

Each example below is a dot on a square. Its position gives two measurements, which we call x₁ (across) and x₂ (up), each between −1 and 1. Its shape gives the right answer, its label: a teal circle or a brass square.

An artificial neuron takes the two measurements, multiplies each by a number called a weight, adds them up, and adds one more number called the bias:

z = w₁x₁ + w₂x₂ + b

It then squashes z into a number between 0 and 1 with the sigmoid function, σ(z) = 1 / (1 + e−z), where e ≈ 2.718 is the base of natural logarithms. Read the result as the neuron’s answer to “how sure am I that this is a circle?” When z is large and positive the answer is close to 1; large and negative, close to 0; and exactly 0.5 when z = 0. The points where z = 0 form a straight line, so a single neuron can only ever split the plane with a straight line. The weights tilt the line, the bias slides it, and bigger weights make the neuron more confident: its soft edge gets sharper.

1.5
−0.5
0.4
Mistakes
8 of 30
Loss (average cross-entropy)
0.575
The line
1.5·x₁ − 0.5·x₂ + 0.4 = 0
Figure 1.One neuron. Drag the line to slide it, drag a handle to turn it, or use the sliders. The line is where the neuron is exactly 50–50; teal shading means it leans towards “circle”, brass towards “square”, and the stronger the colour, the surer it is.

Counting mistakes is the natural score, but it is a poor guide for tuning. Nudge a weight a tiny bit and the count usually doesn’t change at all, so it can’t tell you which way to go. Machine learning uses a smoother score called the loss: a single number that is near 0 for confident right answers and grows the more wrong they are.

The loss used here is cross-entropy. For each example, take the probability the neuron gave to the right answer and work out its negative natural logarithm. A confident right answer (probability 0.9) costs 0.11. A shrug (0.5) costs 0.69. A confident wrong answer, giving the right answer only 0.01, costs 4.6. The loss is the average over all the examples. Because every slight turn of a slider changes every probability slightly, the loss changes smoothly too, and that is what makes the next step possible.

Walking downhill

You tuned the neuron by eye. A computer does it by asking, for each weight: if I increase this number a little, does the loss go up or down, and how steeply? That rate of change is the slope of the loss (with many weights, the list of all the slopes is called the gradient). Then it moves each weight a small step against its slope, downhill:

new w = w − η × slope

The number η (the Greek letter eta) is the learning rate, or step size. It is chosen by people, not learned, and getting it right is one of the main practical chores of training real networks. This method is called gradient descent.

To see the learning rate on its own, the next figure uses the simplest loss there is, a bowl with one weight: L(w) = (w − 3)². Its lowest point is at w = 3, and its slope at any w is 2(w − 3). The teal line shows the slope where the ball sits.

0.10
Weight w
−1.000
Loss L(w)
16.000
Slope
−8.000
Steps taken
0

Creeping downhill from one side.

Figure 2.Gradient descent on a bowl-shaped loss, starting from w = −1. Each step moves the weight by the step size times the slope, in the downhill direction.

Why those particular numbers? On this bowl, each step multiplies the distance from the bottom by (1 − 2η). With η = 0.1 that factor is 0.8, so every step removes a fifth of the remaining distance. At 0.5 the factor is 0, so the first step lands exactly on the bottom. Between 0.5 and 1 the factor is negative but smaller than 1 in size: the ball jumps across the bowl each time, but by less and less. Above 1, each jump is larger than the one before, and the weight runs off to infinity. Real losses are not perfect bowls, but the same three behaviours (too slow, about right, unstable) show up in real training runs.

More neurons bend the line

Some patterns can’t be split by any straight line. The classic one is called XOR (“exclusive or”): circles in the top-right and bottom-left corners, squares in the other two. Any straight line you draw leaves at least one corner on the wrong side.

The fix is to put neurons in front of the neuron. In a hidden layer, several neurons each look at the inputs and draw their own line. Each passes its z through tanh, a squashing function shaped like the sigmoid but running from −1 to 1. A final output neuron then treats their answers as its inputs, weighs them and adds a bias, just as before. Combining several soft-edged lines lets the network carve out corners, stripes and blobs. A network with H hidden units has 4H + 1 adjustable numbers, which we call its parameters.

Finding the slope for every one of those parameters is done by backpropagation, popularised by Rumelhart, Hinton and Williams in 1986. It applies the chain rule of calculus, starting from the loss and working backwards through the layers, so all the slopes come out of a single backward pass. The network below trains this way, live in your browser. At every step it looks at all 120 examples, computes the gradient, and takes one step of gradient descent.

Examples
4 units
0.50
Training steps
0
Loss (average cross-entropy)
—
Mistakes
—
Parameters
17
Random seed
1
Figure 3.A network with two inputs, one hidden layer of tanh units and a sigmoid output, trained by full-batch gradient descent on cross-entropy. The starting weights come from a random-number generator with a fixed seed, so “Reset” replays the same run. Training pauses when the figure is off screen, and if you prefer reduced motion it waits for you to press Train.

Nothing in the code mentions corners, circles or moons. The network is told only the 120 examples and the rule “go downhill on the loss”, and the shape appears. That is all “learning” means here: adjusting numbers so that a formula fits a set of examples.

Memorising is not learning

Getting every training example right sounds like success, but it isn’t the goal. What we want is a network that gets new examples right, ones it has never seen. Real data is also messy: some examples are simply mislabelled.

The next figure trains a network on only 30 examples drawn from the circle pattern, with one label in ten flipped at random to stand for that mess. Alongside, it tests the same network on 400 fresh examples from the same source, which it never trains on. Because a tenth of the fresh labels are flipped too, even a perfect circle would get about 10% of them wrong.

16 units
Training steps
0
Training mistakes (dashed)
—
Fresh mistakes (solid)
—
Fewest fresh mistakes so far
—
Figure 4.The same kind of network as Figure 3 (learning rate 1), trained on 30 examples with 10% of labels flipped, and scored on 400 fresh examples it never trains on. Training stops after 40,000 steps.

This is called overfitting: the network has stopped learning the pattern and started memorising the examples, noise included. With 16 hidden units it has 65 adjustable numbers to fit 30 examples, so it can. In this run, fresh mistakes bottom out at 15% after about 150 steps, then climb to about 37% while training mistakes reach zero. The simplest cure is visible in the last readout: stop training when fresh mistakes are lowest, a trick called early stopping. More data helps most of all.

From 17 numbers to billions

Large language models are trained on the same principle as the small network here: a loss that measures how wrong the guesses are, gradients computed by backpropagation, and many small downhill steps. The differences are of scale and design. For a published example, GPT-3, a language model from 2020, has 175 billion parameters and was trained on 300 billion tokens (pieces of words) of text. Its loss measures how well it predicts the next token. Such models use an architecture called the transformer, which is not shown here.

What this model leaves out

  • Mini-batches. Figure 3 computes the gradient from all 120 examples at every step. Real training uses stochastic gradient descent: each step uses a small random batch, which is noisier but far cheaper when there are billions of examples. Real training also usually uses refinements of plain gradient descent (momentum, Adam).
  • Other activations. We use sigmoid and tanh. Most modern networks use others, such as the ReLU (max(0, z)) and its relatives, which train faster in deep networks.
  • Overfitting, properly. Figure 4 shows overfitting on a toy problem, but not the usual remedies: regularisation (penalising large weights), dropout, data augmentation, and the practice of keeping a separate validation set for choosing when to stop and a test set that is looked at only once. Very large modern networks can also behave in more surprising ways than this small one.
  • Architecture. Real large models have many layers and specialised structures (convolutions for images, transformers with attention for text), not one small hidden layer.
  • The bowl. Figure 2’s loss is a perfect parabola with one weight. Real losses have millions or billions of dimensions, flat regions and many low points. The step size that works best changes during training, and in practice it is often reduced on a schedule.
  • What “learning” means. Here it means fitting a function to examples by adjusting numbers. It says nothing about understanding, and the brain’s neurons work very differently from these.

Sources

  1. Michael A. Nielsen, Neural Networks and Deep Learning (Determination Press, 2015), chapters 1–3. neuralnetworksanddeeplearning.com Sigmoid neurons, gradient descent and the learning rate (ch. 1), backpropagation (ch. 2), and the cross-entropy cost −[y ln a + (1 − y) ln(1 − a)] averaged over examples (ch. 3).
  2. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning (MIT Press, 2016), §4.3 and §6.5. deeplearningbook.org Gradient descent and the effect of the learning rate (§4.3); XOR as the example a linear model cannot learn and a hidden layer can (§6.1); back-propagation (§6.5); stochastic gradient descent (§8.3).
  3. David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, “Learning representations by back-propagating errors”, Nature 323, 533–536 (1986). doi:10.1038/323533a0 The paper that made backpropagation through hidden layers widely known.
  4. Tom B. Brown and others, “Language Models are Few-Shot Learners” (2020). arXiv:2005.14165 GPT-3: 175 billion parameters (abstract); trained on 300 billion tokens (Table 2.1).
  5. Daniel Smilkov, Shan Carter, D. Sculley, Fernanda B. Viégas and Martin Wattenberg, “Direct-Manipulation Visualization of Deep Networks” (2017), describing the TensorFlow Playground. arXiv:1708.03788 Prior art for training a small network live in the browser on two-dimensional toy datasets. Figure 3 is our own design and code.

Colophon

Date
1 October 2026

The gradients in Figure 3 were checked against finite differences (largest relative error 6 × 10⁻⁹). The model code is in nn.js.