Statistical Learning

From probability to generalizing predictors

Prerequisites

graph TD
  Prob[Probability theory] --> Stat[Statistical learning]
  Info[Information theory] --> Stat
  Bayes[Bayesian inference systems] -.->|priors · posteriors| Stat

When to use

Predictive tasks with labeled historical data and measurable loss (regression, classification, ranking).
Generalization matters — you care about performance on unseen examples, not training fit alone.
Foundation before deep learning, RL, and probabilistic infrastructure topics.

When not to use

Pure rule compliance with no learning signal — explicit logic may be required by regulators.
Causal intervention questions — correlation-fit models need causal design, not just loss minimization.
Adversarial inputs without robust training — standard ERM can fail silently.

How to read the diagrams

Bias–variance contrasts underfit, good fit, and overfit hypotheses; risk decomposition shows empirical risk plus complexity penalty from finite data. The ascii overfitting sketch below draws the same three curves.

Simple Fundamental Explanation

Imagine you are trying to guess if a house will sell for more than $500,000.

Rule-Based Programming: You write a rule: IF Square_Feet > 2000 AND Bedrooms > 3 THEN Price > 500k. This is rigid. It fails if the house is a mansion but located in a ghost town.
Statistical Learning: You look at 10,000 past house sales. You plot them on a graph. You notice a trend: bigger houses generally sell for more money, but there is “noise” (some small houses sell for a lot). You draw a “Line of Best Fit” straight through the middle of the noisy cloud of points.

When a new house goes on the market, you look at where its square footage lands on your line, and you read the corresponding price. You didn’t write a rigid rule; you mathematically captured the historical probability pattern.

Statistical Learning Theory is the mathematical framework behind Machine Learning. It focuses on finding predictive functions based on data, while strictly managing the tradeoff between learning the signal and memorizing the noise.

Deep Dive: How It Works Under the Hood

The goal of statistical learning is to estimate an unknown function $f (X) = Y$ that maps inputs $X$ (features) to outputs $Y$ (targets). Because the real world is messy, the true relationship is $Y = f (X) + ϵ$ , where $ϵ$ is random noise.

1. The Model (Hypothesis Space)

You must choose the shape of the line you want to draw.

Linear Regression: A straight line. Simple, but cannot model complex curves.
Decision Trees: Step-functions.
Neural Networks: Highly complex, wavy, multi-dimensional curves.

2. The Loss Function (Error)

To draw the “Best” line, you need to mathematically define “Best.” This is the Loss Function. If your line predicts $400 k, an d t h e h o u seso l df or$ 500k, your error is $100k. The most common loss function is Mean Squared Error (MSE), which squares the errors to heavily penalize massive mistakes.

3. The Bias-Variance Tradeoff

This is the most important concept in all of Machine Learning.

High Bias (Underfitting): You draw a rigid straight line through a curved cloud of dots. The model is too simple. It misses the underlying truth.
High Variance (Overfitting): You draw a crazy, squiggly line that perfectly connects every single dot, capturing all the random noise $ϵ$ . When a new house appears, the squiggly line will make a wildly incorrect prediction.

Statistical learning is the art of finding the mathematical sweet spot where the line captures the true curve, but ignores the random noise.

Visual Diagram: Overfitting

Diagram 1 — Bias–variance model complexity

flowchart LR
  U["High bias<br/>straight line · underfit"]
  G["Sweet spot<br/>captures signal"]
  O["High variance<br/>wiggly · overfit"]
  U --> G --> O

Diagram 2 — Empirical risk and generalization gap

flowchart TB
  Train["Minimize empirical risk<br/>on n samples"]
  Pen["VC complexity penalty<br/>grows with model capacity"]
  True["Bound true expected risk"]
  Train --> Pen --> True

House Prices vs Square Feet

(X = Data point)

[ Underfit (High Bias) ]
   /
  /  X
 / X   X
/  X

[ Good Fit ]
     . -- X
   .  X
 . X
. X

[ Overfit (High Variance) ]
    X
   / \   X
  /   \_/
 X
(The line perfectly hits every point, but it's wildly erratic)

Practical Example: Recommendation Engines

Netflix’s recommendation engine doesn’t have a rigid IF statement linking “Action Movies” to “Sci-Fi”. It uses statistical learning (specifically, Matrix Factorization and Deep Learning).

It takes a giant matrix of Users vs Movies. Most of the matrix is empty (you haven’t watched most movies). The algorithm learns latent statistical features. It figures out probabilistically that Users who watch Movie A and B have a 92% mathematical probability of also liking Movie C. It learned the hidden statistical distribution of human taste, allowing it to predict exactly what you want to watch tonight.

The Mathematics: Equations and In-Depth Analysis

1. Risk Minimization

The theoretical goal is to minimize the Expected Risk $R (f)$ , which is the expected loss over the entire true distribution of data $P (X, Y)$ :

R (f) = E_{X, Y} [L (Y, f (X))]

However, we don’t have the true distribution of the entire universe. We only have our training dataset of $n$ samples. So we calculate the Empirical Risk $\hat{R} (f)$ :

\hat{R} (f) = \frac{1}{n} i = 1 \sum n L (y_{i}, f (x_{i}))

The core problem of ML is that minimizing Empirical Risk (making the loss 0 on your training data) leads directly to Overfitting.

2. Vapnik-Chervonenkis (VC) Dimension

How do we mathematically guarantee that a model trained on $n$ samples will actually generalize to new data? Vapnik and Chervonenkis introduced the VC Dimension $h$ , which measures the “capacity” or complexity of a model (how squiggly the line is allowed to be).

The VC Generalization Bound states that with probability $1 - δ$ , the true Expected Risk is bounded by the Empirical Risk plus a complexity penalty:

R (f) \leq \hat{R} (f) + \frac{h ( ln ( \frac{2 n}{h} ) + 1 ) - ln ( \frac{δ}{4} )}{n}

What this math means:

As you add more data $n$ , the penalty term shrinks. Big data makes the model more reliable.
As you increase model complexity $h$ (e.g., adding more layers to a neural network), the penalty term grows.
To prevent the penalty (and thus the true Error) from exploding, you must use Regularization (mathematically penalizing complex weights during training, such as L1/L2 regularization) to artificially keep $h$ low while minimizing $\hat{R} (f)$ .

Lab

Bias–variance tradeoff: fit noisy quadratics with increasing polynomial degree.

Model degreedegree 1

Does very high degree usually overfit (low train MSE, higher test MSE)?