Classification and probability

Is this email spam? Is this tumor benign? Which digit is this? These are ClassificationA prediction task where the answer is one of a fixed set of categories, such as spam or not spam, or which digit a drawing shows.Open in glossary problems: the answer is a category, not a number. A good classifier does more than name a category. It says how sure it is, because “90% likely to be spam” and “51% likely to be spam” call for different actions.

From a score to a probability

Start with something you already know: a weighted sum of the inputs, like linear regression, which gives a score $z = w\,x + b$ . The score can be any number, from very negative to very positive. A probability must stay between 0 and 1. The SigmoidThe S-shaped function 1 / (1 + e^-z), which squashes any number into the range 0 to 1 so it can be read as a probability.Open in glossary function bridges the two:

\sigma(z) = \frac{1}{1 + e^{-z}}

Large positive scores map to probabilities near 1, large negative scores to probabilities near 0, and a score of exactly 0 maps to 0.5. Putting a weighted sum through a sigmoid gives Logistic regressionA classifier that computes a weighted sum of the inputs and passes it through the sigmoid to get the probability of the positive class. Despite the name, it is used for classification.Open in glossary. Despite the name, it is a classifier.

From score to probability

Made-up data: hours of practice before a driving test, and whether the test was passed. The curve is the model's probability of passing.

Failed (y = 0)
Passed (y = 1)
Gap to the right answer (thickest: the costliest point)

Weight w (steepness)0.40

Bias b (shift)-1

Mean cross-entropy0.378

Accuracy at 50%81%

50% point2.5 hours

Try this

Drag the weight up. The curve gets steeper: the model becomes more decisive about the change from fail to pass.
Drag the bias. The whole curve slides left or right, moving the 50% point.
Find the setting with the lowest mean cross-entropy you can, then press Train (Adam) and see how close you got.
Make the curve very steep in the wrong place. The thick orange gap marks the costliest point, and the loss jumps.

Measuring a probabilistic mistake

Squared error is a poor fit for probabilities. What we care about is how much probability the model gave to the answer that turned out to be right. The standard loss is Cross-entropyA loss for predicted probabilities: minus the log of the probability the model gave to the right answer. Confident wrong answers are punished very heavily.Open in glossary: minus the logarithm of that probability.

\text{loss} = -\ln p_{\text{correct}}

Probability given to the right answer	Loss
0.99	0.01
0.9	0.11
0.5	0.69
0.1	2.30
0.01	4.61

A confident right answer costs almost nothing, a coin flip costs about 0.69, and a confident wrong answer costs a lot. The loss keeps growing without limit as the probability for the right answer approaches zero, so a model is strongly discouraged from being sure and wrong. One way to read it: cross-entropy measures how surprised the model was by the truth.

Why the gradient is so simpleOptional

For a label $y \in \{0, 1\}$ and predicted probability $p = \sigma(z)$ , the loss on one example is

\ell = -\left[\,y \ln p + (1 - y)\ln(1 - p)\,\right].

Using the fact that $\sigma'(z) = \sigma(z)\,(1 - \sigma(z))$ , the derivative with respect to the score simplifies to

\frac{\partial \ell}{\partial z} = p - y.

So the gradient for each weight is just (predicted probability minus label) times that weight’s input. A point the model gets right with confidence contributes almost nothing; a point it gets badly wrong pushes hard. The demos compute exactly this, averaged over the data.

Two inputs: the decision boundary

With two inputs, the score is $z = w_1 x_1 + w_2 x_2 + b$ . Each point on the plane gets a probability, and the classifier predicts class B wherever that probability is above 50%. The line where it is exactly 50%, where $z = 0$ , is the Decision boundaryThe line or surface where a classifier switches from predicting one class to another.Open in glossary.

Logistic regression

Shading shows the model's probability for each spot: blue for class A, orange for class B, gray where it is unsure. Tap to add points.

Class A
Class B
Decision boundary (50%)
10% and 90% lines

Step size (learning rate)0.50

Data

SeparatedOverlapping

Tap adds

Class AClass B

Steps0

Mean cross-entropy0.6931

Accuracy50%

Weights w1, w2, b0, 0, 0

Try this

Press Train. The boundary swings into place between the two clouds, and the gray band of uncertainty narrows.
Keep training on the Separated data. The boundary barely moves, but the dashed 10% and 90% lines keep squeezing toward it: the model grows ever more confident.
Switch to Overlapping. Training settles at a loss well above zero, and the uncertain band stays wide, because some points really are ambiguous.
Add a few class B points deep inside the blue region and train again. The boundary tilts to accommodate them, but a straight line can only do so much.
Raise the step size to the maximum. On this problem training still converges, just in bigger jumps; try it on the overlapping data too.

The boundary of logistic regression is always a straight line, or a flat plane with more inputs, because the probability depends only on a weighted sum. When the classes are arranged in a ring or a spiral, no straight line separates them. You can fix that by inventing extra input features, such as $x_1^2$ , or by letting a neural network learn its own features, which is what the neural networks track is about.

Key ideas

A classifier outputs probabilities, not just labels, so you know how sure it is.
The sigmoid squashes any score into a probability between 0 and 1. Logistic regression is the sigmoid of a weighted sum.
Cross-entropy is minus the log of the probability given to the right answer: cheap when confident and right, very expensive when confident and wrong.
The decision boundary of logistic regression is a straight line where the probability is 50%.
On perfectly separable data, the weights grow without limit unless regularization stops them.