Machine learning: fitting, grouping, and knowing when it has memorised
Supervised and unsupervised learning, side by side and small enough to see. In the regression panel you have known answers, and the model fits a line that minimises its error against them — move the points and watch the line chase. In the clustering panel there are no answers at all; the algorithm partitions the space by proximity, and you choose how many groups it looks for. Change that number and clusters appear that mean nothing. That is the lesson worth carrying: clustering will always return groups, and whether they correspond to anything real is a question the algorithm cannot answer for you. Then three panels about being wrong. Two classifiers — nearest neighbours and a decision tree — are scored twice over, once on the points they were shown and once on points they were not, and the distance between those two numbers is overfitting made visible: at k = 1 the model is flawless on what it has seen and much worse on everything else, because it memorised rather than learned. Two more take the ideas apart. Logistic regression shows where a probability comes from — a straight line squashed into an S — and that moving the threshold does not change the curve at all, because the model and the decision are separate things and only one of them was trained. And principal components shows two columns that are really one: when measurements move together, most of what distinguishes your rows lies along a single direction, and reducing to it throws away far less than it sounds. The last panel is the one that decides whether a model is any use in practice. A classifier returns a score, not a verdict; somebody has to choose the cut-off, and every choice trades missed cases against false alarms. Move it and watch precision and recall pull in opposite directions — and watch overall accuracy stay high while the model catches almost nobody, which is exactly how a rare-event model gets signed off and then fails in production.
Supervised learning
Linear regression
Regression finds the straight line that best predicts y from x. Add noise and watch how confidently the line still captures the trend — measured by R².
Spread of points around the true line.
- Push noise to 0%. Every point sits on the line and R² is 1. This is what a perfect fit looks like, and real data never looks like it.
- Now push noise to 40% and leave the dataset small. R² collapses, but the line still points the right way. A low R² means the scatter is wide, not that the trend is wrong.
- Keep the noise and drag the dataset up to 150. The line steadies while R² barely moves. More data buys you confidence in the line, not a tighter cloud.
The line minimizes squared error. More noise lowers R² but the fit stays unbiased.
Unsupervised learning
k-means clustering
With no labels, k-means groups points by closeness. Pick how many clusters to look for, then step through the algorithm as centroids settle into the data.
How many groups to find.
- Set k to 3 and press Step slowly. Points recolour first, then the centres move. Those two actions, alternating, are the whole algorithm.
- Re-seed a few times at the same k. You will not always get the same answer. Where it starts changes where it ends — which is why real runs are repeated.
- Now set k to 6. It confidently returns six groups. Nothing in the algorithm can tell you they are not real — only you can.
Each step reassigns points to the nearest centroid, then moves centroids to their mean — until nothing changes.
Classification
k-nearest neighbours
To label a new point, look at the k closest points you already know and take the majority vote. The shaded background is the model's answer for every possible point.
How many known points get a vote.
How much the two groups bleed into each other.
- Leave k at 1 and read both bars. 100% on points it was shown, far less on points it was not. It stored the answers rather than learning the pattern.
- Raise k to about 9. The boundary smooths and the two bars converge. The model got worse at the data it has and better at everything else.
- Push k to 25, then raise the overlap. Too much smoothing erases the real shape. There is a best k, and it is neither end of the slider.
At k = 1 the model is perfect on data it has seen and worse on data it has not — it memorised. Raising k smooths the boundary and closes that gap, until it smooths away the real shape too.
Classification
Decision tree
A tree asks one question at a time — is x above this line, is y below that one — carving the space into rectangles. Deeper means more questions.
How many questions deep it may go.
How much the two groups bleed into each other.
- Start at depth 1. One straight cut across the whole space — a single question. It is already about 80% right.
- Step up to 3, then 4. Both bars rise together. Everything the tree learns here is real structure, and it helps on unseen points too.
- Now go to 8 or 9 and watch the regions. Thin slivers appear around single points. The seen bar reaches 100% while the unseen bar falls — the tree is describing this data, not the world.
Every extra level fits the data you have a little better and the data you do not have a little worse. Watch the two accuracies separate — that gap is overfitting, drawn.
Evaluating a model
Where you set the threshold
A classifier gives a score, not a verdict. You choose the cut-off. Move it and watch which mistakes you trade for which — there is no setting with none of either.
Score above this is called positive.
Share of cases that really are positive.
- Drag the threshold to 15%. Recall hits 100% — nothing is missed — and precision collapses. You catch everyone by accusing everyone.
- Now drag it to 85%. Precision reaches 100% and recall collapses. Every flag is right because you barely flag anything.
- Set the positive case to 2% and watch accuracy alone. Accuracy stays high at almost any threshold while recall is dreadful. This is how a rare-event model gets approved and then fails.
Accuracy hides this. When only a few cases are positive, calling everything negative scores well and catches nobody — which is why precision and recall are quoted instead.
Classification
From a score to a probability
A straight line can predict a number, but not a yes or no — it runs past 1 and below 0. Logistic regression squashes the line into an S, so every input becomes a probability between them.
Clear cases at the ends, ambiguous ones in the middle.
- Set separability low. The S flattens towards a straight, shallow slope. An unsure model says 0.5 to almost everything.
- Raise separability to the top. The curve becomes almost a step. Confidence is the steepness — the model learned it, you did not set it.
- Now move the threshold without touching anything else. The curve does not move at all. The model and the decision are two different things, and only one of them was trained.
The curve is the model. The threshold is a separate decision made afterwards, by a person — which is why the same trained model can be cautious or aggressive without retraining anything.
Unsupervised learning
Two columns that are really one
When two measurements move together, most of what distinguishes your rows lies along a single direction. The line is that direction; the ticks are where each point lands once you drop everything else.
From independent to nearly identical.
- Set the two columns to barely related. The cloud is round and the line could point anywhere. One direction keeps only about half the information.
- Raise the relationship to the top. The cloud becomes a cigar and one direction keeps almost everything. The second column was telling you what the first already had.
- Turn on the projection at a high setting. Every point slides onto the line and the picture barely changes. That is two columns becoming one, with almost nothing lost.
This is what dimensionality reduction is: not throwing columns away, but finding the directions the data actually uses. Two correlated columns carry barely more information than one.