Build a brain
from scratch.

Strip away the biology metaphors. As an engineer, hold these three facts and you can derive the rest.

1. It's a function. A network takes an input vector x and returns an output y_hat. Internally it's just multiplications, additions, and a sprinkle of simple nonlinear functions. Nothing you couldn't write in an afternoon.

2. It has parameters. Scattered through that function are numbers: the weights and biases. These are the knobs. "Training" means finding knob settings that make y_hat match reality on examples we've seen.

3. It's differentiable. Because every operation inside is smooth, we can ask: "if I nudge this one knob up a hair, does the error go up or down, and how fast?" That answer is a gradient. Collect the gradient for every knob and you know exactly which way to step. Repeat a few thousand times and the function fits the data.

The rest of this guide is just zooming into each of those words (neuron, activation, loss, gradient, backprop) until you've built the machine yourself and watched it learn.

A single neuron does three tiny things, in order: it weights its inputs, sums them with a bias, and passes the result through a nonlinear "activation."

Think of each weight w as a volume knob on one input: how much should this feature matter, and with what sign? The bias b shifts the whole sum: it lets the neuron fire even when inputs are zero, or stay quiet until inputs are large. The activation bends the straight line into a curve (rung 02 explains why that matters).

The two views below are the same neuron. Left: the activation curve, where your current z lands on σ. Right: the input plane, shaded by output, with the neuron's decision line (z = 0) and the weight vector drawn as an arrow. Move the weights and watch the line rotate to face the w direction; move the bias and watch it slide.

Stack two linear layers and you get one linear layer. The activation function is what makes depth meaningful.

No matter how many linear layers you chain, the composition is still a single straight-line transform. It can only ever draw a flat decision boundary. Real data (spirals, clusters, the curve of a sine) is not separable by a line.

Insert a nonlinearity between layers and the spell breaks. Each layer can now bend space, and bends stacked on bends produce arbitrarily complex shapes. This is the informal content of the universal approximation theorem: a network with one hidden layer and a nonlinearity can approximate essentially any continuous function, given enough neurons.

The usual activations

You already saw all three live in rung 01: flip the dropdown there and notice ReLU's hard hinge versus tanh's gentle S. That shape is the only "intelligence" a single neuron has; everything else is composition.

Here's the theorem made visible. Each neuron in a hidden layer produces one bump or hinge; adding them up builds a custom curve. Below, a one-hidden-layer network fits a wiggly target. Watch the faint individual neuron outputs (the building blocks) combine into the bold curve that chases the target. Add neurons and the fit gets sharper: that's "given enough neurons, approximate anything."

A layer is just many neurons looking at the same inputs in parallel. Stack the weights into a matrix and the whole layer becomes a single matrix multiply, which is exactly why GPUs make this fast.

A multilayer perceptron (MLP) chains these: the activations of one layer are the inputs to the next. Running data left-to-right through every layer to produce a prediction is the forward pass.

import numpy as np

def relu(z): return np.maximum(0, z)

def forward(x, params):
    # params is a list of (W, b) tuples, one per layer
    a = x
    for i, (W, b) in enumerate(params):
        z = W @ a + b              # weighted sum (the matrix multiply)
        a = relu(z) if i < len(params)-1 else z  # last layer: no squish
    return a                          # the prediction y_hat

Step through it below. Each highlighted row of W is one neuron; its dot product with the input x (plus that neuron's bias) produces one entry of the output z. Four neurons, four rows, four outputs: all computed in one multiply.

To improve, the network needs a single scalar that says "how bad was that prediction?" Smaller is better. That scalar is the loss. Training is literally the search for parameters that minimize it.

Regression → mean squared error

Why squared? The error ŷ − y is a length; squaring it is literally the area of a square with that side. Double the miss and the penalty quadruples, so MSE cares far more about fixing big mistakes than small ones. Drag the prediction and watch the square (the loss) grow.

Classification → cross-entropy

For one example the formula collapses to a single term: the loss is just −log(p), where p is the probability the model assigned to the correct answer. Get the truth confidently right (p→1) and loss vanishes. Be confidently wrong (p→0) and −log(p) rockets toward infinity. That asymmetry is the whole point: it makes overconfident mistakes very expensive.

The loss is the only thing the network is trying to reduce. Choosing it is a design decision with real consequences: it defines what "good" means. MSE says "be close on average." Cross-entropy says "be calibrated and don't be confidently wrong."

We have a loss landscape and we want the lowest valley. We can't see the whole map, but at our current spot we can compute the slope. The gradient is the vector of slopes, one per parameter, pointing in the direction of steepest increase. So we step the opposite way.

Read it as: "to update a weight, subtract a small multiple of its gradient." If nudging w up makes the loss go up (positive gradient), we move w down. If up makes loss go down (negative gradient), the minus sign moves w up. Either way, toward lower loss.

The learning rate η is the single most important dial you'll touch. Too small and learning crawls. Too large and you overshoot the valley and bounce, or diverge entirely. Drag the slider below and watch the ball.

One question remains, and it's the crux of the whole field: the network has thousands of weights buried across many layers. How do we get ∂L/∂w for every single one, efficiently? That's backpropagation.

Backprop is calculus's chain rule, applied with bookkeeping. The forward pass builds an expression out of small operations. Backprop walks that expression in reverse, handing each operation its share of the blame for the final loss.

Here is the only calculus you need:

Every operation knows its own local derivative (a multiply's, an add's, a tanh's, each is a one-liner). Backprop just multiplies these local derivatives along every path from the loss back to each weight. Each node receives gradient from its output, multiplies by its local derivative, and passes the result to its inputs. A single backward sweep computes ∂L/∂w for all weights at once.

Below is a real computational graph for one neuron feeding a squared-error loss. Press Forward to compute every value left-to-right (cyan). Then press Backward to flow gradients right-to-left (amber), one step at a time, with the chain rule spelled out. Tweak the inputs and run it again.

Notice the payoff: the gradient on w₁ tells you exactly how to change w₁ to reduce the loss, and you got it for free, just by reversing the same graph you already built going forward. That's the trick that scales to GPT.

You don't want to derive gradients by hand for every new model. Instead, build an engine that records operations as they happen and knows how to reverse each one. This is "automatic differentiation": the beating heart of PyTorch and JAX, and you can write a usable version in ~40 lines.

The trick: wrap every number in a Value object that remembers (a) its data, (b) which values produced it, and (c) a closure that knows how to push gradient to those parents. Calling .backward() on the final loss walks the graph in reverse topological order and fills in .grad everywhere.

import math

class Value:
    def __init__(self, data, _children=()):
        self.data = data
        self.grad = 0.0            # ∂L/∂self, filled in by backward()
        self._backward = lambda: None  # local rule: push grad to parents
        self._prev = set(_children)

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other))
        def _backward():
            self.grad  += 1.0 * out.grad   # + passes grad through
            other.grad += 1.0 * out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other))
        def _backward():
            self.grad  += other.data * out.grad  # × sends the OTHER input
            other.grad += self.data  * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,))
        def _backward():
            self.grad += (1 - t*t) * out.grad   # d/dz tanh = 1 - tanh²
        out._backward = _backward
        return out

    def backward(self):
        # build reverse topological order, then apply each local rule
        topo, seen = [], set()
        def build(v):
            if v not in seen:
                seen.add(v)
                for p in v._prev: build(p)
                topo.append(v)
        build(self)
        self.grad = 1.0              # ∂L/∂L = 1, the seed
        for v in reversed(topo):
            v._backward()

That's the entire idea. With __add__, __mul__, and tanh, you can build the neuron from rung 06: a = (w1*x1 + w2*x2 + b).tanh(), call loss.backward(), and read w1.grad directly. The same values the scope visualized, computed by code.

Watch the engine run. The expression L = tanh(w·x + b) uses exactly the three ops above. Record plays the forward pass: each operation builds a new Value, computes its .data, and pushes itself onto the tape (the topological order). Replay runs backward(): it seeds ∂L/∂L = 1 and walks the tape in reverse, firing each node's _backward closure to hand gradient to its parents. Each step echoes the exact closure from the code above as it fires.

Everything assembled. The training loop is shorter than its explanation, and once you've seen it, you'll recognize it inside every deep-learning codebase on earth.

for step in range(epochs):
    y_hat = model(x)              # 1. FORWARD: predict
    loss  = loss_fn(y_hat, y)     # 2. SCORE: how wrong?

    for p in model.parameters():
        p.grad = 0.0             # 3a. reset grads (they accumulate!)
    loss.backward()               # 3b. BACKWARD: fill every .grad

    for p in model.parameters():
        p.data -= lr * p.grad     # 4. UPDATE: step every knob downhill

Here is that loop turning. Each pass lights the four stages in order while a genuine tiny network (1 → 6 → 1, fitting a curve) takes one real gradient step. The loss on the right ticks down as the cycles accumulate.

That loop is what's running, thousands of times a second, in the live demo on the next rung. Go watch it actually learn.

This is the hero moment. Below is a genuine multilayer perceptron (forward pass, cross-entropy loss, backprop, gradient-descent update, exactly the loop from rung 08) training in your browser on a 2D classification problem. The background shading is the network's current decision surface; watch it bend itself around the data as the weights learn.

Pick a dataset that isn't linearly separable (spiral is the cruel one). Hit Train. Then experiment: drop the hidden layers to a tiny width and watch it fail to find the shape; crank the learning rate too high and watch the loss explode; switch ReLU vs tanh and feel the difference. Every mechanism in this guide is running here at once.

The math is simple; getting it to train well is craft. The knobs below are where most of your real time will go.

Weight initialization

Start all weights at zero and every neuron in a layer computes the same thing and receives the same gradient: they never differentiate. Start them too large and activations saturate (tanh flatlines, gradients vanish). The fix is principled random init scaled to layer size: He initialization for ReLU, Xavier/Glorot for tanh. The live demo uses these; that's why it trains instead of stalling.

Learning rate & schedules

Still the highest-leverage dial. In practice you don't keep it fixed: you warm up from small, then decay over training. Adaptive optimizers like Adam give each parameter its own effective rate using running averages of its gradients, usually the sane default over plain gradient descent.

Batches & stochasticity

Computing the loss over the entire dataset every step is expensive and, surprisingly, worse. Mini-batches (32–512 examples) give a noisy gradient estimate, and that noise helps escape bad valleys. One pass over all the data is an epoch.

Both paths below are real gradient descent on the same real dataset (a small linear-regression problem, so the loss over the two weights is an exact bowl you can see). Full-batch averages the gradient over every point: a smooth, direct glide to the minimum. A small mini-batch estimates the gradient from a handful of points, so its path jitters. Shrink the batch and watch the noise grow.

Overfitting & regularization

A big network can memorize the training set and fail on anything new. You catch this by watching a held-out validation set diverge from training loss. Defenses: more data, dropout (randomly silence neurons during training), weight decay (penalize large weights), and early stopping.

See it happen. The widget trains a real MLP (1 → H → 1) on a handful of noisy points sampled from a smooth curve, and tracks its loss on both the training points and a held-out validation set. Early on both fall together. Then the training loss keeps dropping toward zero while the validation loss bottoms out and climbs: the network is memorizing noise. Widen the capacity or crank the noise to make the gap yawn; narrow it and the gap closes.

Vanishing & exploding gradients

In deep stacks, repeated multiplication of small local derivatives can drive gradients to zero (nothing learns) or blow them up (everything diverges). The historical answers, ReLU, careful init, normalization layers (BatchNorm, LayerNorm), and residual connections (let signal skip layers), are precisely what made very deep networks trainable.

You can see it. Below, a signal flows forward through 12 layers and a gradient flows backward through them. The bars are the magnitude (RMS) at each layer. Slide the weight scale, or press Sweep to watch it pass through every regime: too small and both the signal and the gradient decay to nothing (vanishing); too large and they blow up (exploding). The narrow healthy band in the middle is exactly what He/Xavier init computes for you automatically.

Everything from here is the same skeleton (forward, loss, backward, update) with smarter structure for the forward pass. Each advance is essentially a clever answer to "what should the layers compute?"

How to actually become a hero

Build the scalar autograd from rung 07 yourself, by hand, no copy-paste. Then make it train the MLP from rung 09 on real data. Re-derive backprop for a 2-layer net on paper once; it cements the chain rule permanently. Then port it to PyTorch and feel how loss.backward() is the exact engine you built. Finally, implement attention from scratch; it's smaller than you'd guess and it's the gateway to everything current.