What is overfitting?
Overfitting happens when a machine learning model learns its training data too closely. Instead of capturing the general pattern, it memorises the specific examples, including their random noise and quirks. The result looks great on the data it was trained on and then performs poorly on new, unseen data. That gap between training performance and real-world performance is the telltale sign.
The opposite problem is underfitting, where the model is too simple to capture the pattern at all and does badly everywhere. Good training lives in between: complex enough to learn the real signal, restrained enough to ignore the noise.
In plain words
Imagine a student who memorises last year's exam answers word for word. They ace a practice run on those exact questions, but the moment the real exam asks something slightly different, they are lost. They learned the answers, not the subject. An overfitted model does the same: it memorised the examples instead of learning the underlying rule.
Why it matters
- It hides behind good numbers. A model can show 99% accuracy in testing and still fail in production, because it was measured on the data it memorised.
- It wastes the investment. A model that does not generalise is not usable, no matter how much compute went into training it.
- It is the core reason for a proper test set. You only trust a model after checking it on data it has never seen.
Common pitfalls
- Testing on training data. If you measure performance on the same data the model learned from, overfitting stays invisible. Always hold out a separate test set.
- Too little data, too complex a model. A large model with few examples will memorise them. More data or a simpler model usually helps.
- Training too long. Past a point, extra training stops learning the pattern and starts learning the noise. Early stopping prevents this.
- Ignoring regularisation. Techniques like dropout, weight penalties, and cross-validation exist specifically to keep models from memorising. Skipping them invites overfitting.
Related articles:
- What is gradient descent? - The algorithm that does the actual learning, and where it can overshoot.
- What is a neural network? - The kind of model most prone to overfitting without care.
- What is fine-tuning? - Adapting a model on new data, where overfitting is a common risk.
Want to stay one step ahead?
Don't miss our best insights. No spam, just practical analyses, invitations to exclusive events, and podcast summaries delivered straight to your inbox.