Transformer (GPT Architecture)

Definition

The transformer is the neural-network architecture behind modern LLMs. A GPT is a decoder-only transformer (the encoder and cross-attention of the original “Attention Is All You Need” model are removed). At a high level it has three parts: text + positional embeddings → a stack of transformer decoder blocks → a projection to vocabulary. Its success comes less from the architecture itself than from being highly parallelizable on GPUs and scaling well with data and parameters.

---

Core Ideas

Embeddings

• Token embeddings — a learned [n_vocab, n_embd] lookup table (wte); token IDs alone are poor inputs (their magnitudes imply false relationships).
• Positional embeddings — a learned [n_ctx, n_embd] table (wpe) that injects order, since attention is otherwise position-agnostic. This caps input length at n_ctx.
• The two are summed so each position carries both word and position information.

Decoder block

Each block has two sublayers, each wrapped with a residual connection and pre-norm layer normalization (x + sublayer(layer_norm(x)), the GPT-2 arrangement):

1. Multi-head causal self-attention — the only place inputs communicate.
2. Position-wise feed-forward network (FFN) — a 2-layer MLP that projects up to 4*n_embd and back down; despite attention’s fame, ~80% of GPT-3’s parameters live here.

Stacking n_layer blocks sets the model’s depth; n_embd sets its width (GPT-3: 96 layers, 12288 embedding).

Attention, built up

• Attention — scaled dot-product: softmax(QKᵀ/√d_k)·V.
• Self — Q, K, V all come from the same sequence (via learned projections), letting a word like “he” attend to “Jay”.
• Causal — a mask sets positions j > i to -∞ before softmax so a token can’t “see the future” (implemented as (1 - tri(n_seq)) * -1e10).
• Multi-head — split Q/K/V into n_head heads of dimension n_embd/n_head, attend per head, then concatenate — giving the model multiple relationship subspaces.

Supporting layers

• GELU — the smooth activation used instead of ReLU.
• Layer normalization — standardizes to mean 0, variance 1, then scales/offsets with learnable γ, β (preferred over batch norm in transformers).
• Linear / projection — matrix multiply + bias, x @ w + b.

Projection to vocab (the LM head)

A final layer norm, then multiply by wteᵀ to produce logits over the vocabulary (softmax is left off for numerical/flexibility reasons). This “language-modeling head” can be swapped for a classification head when fine-tuning.

Fine-tuning

• Classification fine-tuning — replace the LM head with a class projection on the last token.
• Generative fine-tuning — language-model the input concatenated with the label (e.g. article + summary).
• Instruction / supervised fine-tuning — train on human-labeled instruction+completion pairs; this is a form of AI alignment.
• Parameter-efficient fine-tuning — adapters / freezing to update a small fraction of parameters.

---

Relationships

• Neural Network — the perceptron-to-CNN foundation transformers build on
• Machine Learning — training via gradient descent on a loss function
• Retrieval-Augmented Generation (RAG) — grounds transformer LLM outputs with external knowledge
• AI & Machine Learning — parent topic
• AI Inference Hardware — the silicon shaped around serving this architecture, and the KV-cache movement it tries to avoid