Logistic Regression

A text classifier built on logistic regression, implemented from scratch. Where naive Bayes is generative — it models how documents are produced — logistic regression is discriminative: it learns a decision boundary directly, weighting features by how much each one moves the answer.

The model runs a linear score through a sigmoid. Each document becomes a feature vector $\mathbf{x}$ (word counts and indicators). The model scores it with a weight vector $\mathbf{w}$ and bias $b$, then squashes the score to a probability with the logistic function:

The output lies in $(0,1)$ and is read as the probability of the positive class.

The decision rule thresholds that probability at $0.5$ to get the label. Because $\sigma (z)\ge 0.5$ exactly when $z\ge 0$, the boundary is the hyperplane $\mathbf{w}\cdot \mathbf{x}+b=0$: the classifier is linear even though the probability it reports is not.

Training uses cross-entropy as its loss, maximizing the likelihood of the training labels, equivalently minimizing the average negative log-likelihood. For a single example the loss is

This penalizes a confident wrong prediction sharply and a correct one lightly. It is convex in $\mathbf{w}$, so gradient descent reaches the global optimum.

Squared error $(\hat{y}-y{)}^2$ would also measure the miss, but paired with a sigmoid it is non-convex in $\mathbf{w}$, and its gradient carries a factor of ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$. That factor collapses toward zero whenever the model is saturated — confidently near $0$ or $1$ — so a confidently wrong prediction produces almost no gradient and learning stalls exactly where it should correct hardest. Cross-entropy is designed to cancel that factor.

The reward is a clean gradient: with cross-entropy the ${\sigma }^{\prime }$ term cancels, and the gradient with respect to the weights is just the prediction error times the features:

Descent repeats $\mathbf{w}\leftarrow \mathbf{w}-\eta (\hat{y}-y)\mathbf{x}$ over the training set until the loss stops falling. An $L_2$ penalty $\frac{\lambda }{2}\Vert \mathbf{w}{\Vert }_2^2$ on the weights adds a $\lambda \mathbf{w}$ term to the gradient, keeping weights from blowing up on rare features that appear in only one class. Multi-class problems use the softmax generalization, one weight vector per class.

You can read the full report.