Generative Pre-trained Transformer

A character-level GPT implemented from scratch in PyTorch — every module written out by hand, no nn.Transformer shortcuts. It trains on a corpus of 2023 AI and technology writing (the siavava/ai-tech-articles dataset, filtered to that year), building its vocabulary from the sorted set of characters in the text, and generates one character at a time.

Attention works as a soft lookup: each token emits a query, a key, and a value, and attends to the others by how well its query matches their keys:

with the $\sqrt{d_k}$ keeping dot products in softmax's well-behaved range. In code a Head is three bias-free nn.Linear projections into a head_size subspace, a registered lower-triangular tril buffer for the causal mask that zeroes attention to future positions, and dropout on the weights. MultiHeadAttention runs six such heads in parallel ($\text{head\_size}=384/6=64$) and projects their concatenation back to the model width, so different heads track different relationships.

A Block is pre-norm: x = x + sa(ln1(x)) then x = x + ffwd(ln2(x)), where FeedFoward widens to four times the model dimension through a ReLU and back. GPTLanguageModel stacks six of these in an nn.Sequential, fronted by a token embedding table and a learned position embedding table — attention alone is permutation-invariant, so the positions supply order — and closed by a final LayerNorm and an lm_head linear tie to the vocabulary. Weights initialize from a normal with standard deviation 0.02.

The trained configuration (config.yml) is small enough to run on one GPU:

• context 256 tokens, model width 384, heads 6, layers 6.
• dropout 0.2, batch 32, AdamW at learning rate $3\times 10^{-4}$.
• 10,000 iterations, with train and validation loss estimated over 200 batches every 500 steps.

Training minimizes cross-entropy

and generate samples autoregressively — crop the context to the last 256 tokens, softmax the final logits, draw one character with torch.multinomial, append it, and repeat. A FastAPI layer (main.py) loads a saved checkpoint and exposes GET /api/{query}, returning 150 generated characters as JSON.