Neural Networks & Word Embeddings

A feedforward neural network in PyTorch for multi-class classification, paired with an inspection of the word embeddings it learns — measuring closeness with cosine similarity and projecting the high-dimensional space to a plane with t-SNE.

The classifier is a network of linear layers and nonlinearities, mapping an input to class scores by alternating affine maps with an elementwise nonlinearity. With one hidden layer,

where $\varphi$ is a nonlinearity such as ReLU. The nonlinearity is what lets the network represent functions a single linear layer cannot; stacked layers compose into richer decision boundaries.

The output scores $\mathbf{z}$ become a distribution over the $K$ classes with softmax, and training minimizes the cross-entropy against the true label:

PyTorch handles the gradients by backpropagation; the weights update by gradient descent over minibatches.

The softmax carries a cost. That denominator sums $e^{z_j}$ over every class, which is cheap for a handful of labels but ruinous when the "classes" are a whole vocabulary, as in the skip-gram objectives that learn general-purpose embeddings. Negative sampling sidesteps the sum: each real (word, context) pair is trained against a few randomly drawn negative pairs, and one giant normalization collapses into a handful of binary present-or-absent decisions. The vectors that fall out carry the same geometry, learned at a fraction of the cost.

Embeddings carry meaning as geometry. The first layer maps each word to a dense vector, trained end to end. After training, related words sit close together, and closeness is measured by the angle between vectors rather than their length:

Cosine similarity ignores magnitude, so frequent and rare words compare on equal footing. t-SNE then compresses the embedding space to two dimensions for plotting, preserving local neighborhoods so clusters of related words show up visually — a qualitative check that the model learned structure and not noise.

The clusters are the payoff: words with similar usage land near each other, and the parallelogram shows that a relation like gender is a single shared direction, so the space encodes structure, not just proximity.

You can read the full report.