Linear Regression Classifiers

Linear regression with regularization, comparing ridge and lasso on the same data. Both add a penalty on the coefficient magnitudes to the least-squares objective; the difference in the penalty's shape changes what the fitted model looks like.

Both share the same fit term and differ only in the penalty. Ordinary least squares minimizes squared error alone, which overfits when features are many or correlated. Ridge and lasso add a norm penalty scaled by $\lambda$:

Ridge penalizes the sum of squared coefficients ($L_2$); lasso penalizes the sum of absolute values ($L_1$). In both, $\lambda$ trades fit against model complexity: $\lambda =0$ recovers plain least squares, and larger $\lambda$ shrinks the coefficients further toward zero.

Whether a penalty zeros coefficients comes down to the geometry of its constraint region. Each penalized objective has an equivalent constrained form: minimize the squared error subject to $\Vert \mathbf{w}\Vert \le t$, with $t$ set by $\lambda$. The squared-error term draws elliptical contours around the unconstrained least-squares solution, and the fit is the point where the smallest contour first touches the feasible region. That region's shape decides where they meet.

The $L_1$ region is a diamond with corners on the axes, and expanding ellipses tend to first touch it at a corner — where some coordinate is exactly zero. The $L_2$ region is a smooth ball with no corners, so the contour touches at a generic point off the axes and ridge shrinks every coefficient without setting any to zero. Ridge keeps all features with small weights; lasso performs feature selection, producing a sparse model that names the few features that matter.

Ridge is differentiable everywhere, so setting its gradient to zero,

gives a closed form; the added $\lambda I$ also makes the inverse stable when $X^{\top }X$ is near-singular. Lasso has no closed form because $\Vert \mathbf{w}{\Vert }_1$ is not differentiable at zero — the very kink that produces the sparse corner — so it is solved iteratively with coordinate descent or a subgradient method.

The penalty strength $\lambda$ is a hyperparameter, tuned by cross-validation: fit at a grid of $\lambda$ values, score each on held-out folds, and keep the one that generalizes best.