Large Language Models

Next-token probability at transformer scale

Prerequisites

graph TD
  Prob[Probability theory] --> LLM[Large language models]
  Stat[Statistical learning] --> LLM
  Attn[Transformer attention] --> LLM
  Info[Information theory] -.->|entropy · sampling| LLM

When to use

Open-ended language tasks (dialogue, codegen, summarization) where rules cannot enumerate outputs.
Few-shot adaptation via prompting instead of retraining per task.
Probabilistic products that need creativity knobs (temperature) or calibrated uncertainty layers.

When not to use

Hard guarantees (exact arithmetic, legal determinism) without tool use or symbolic checks.
Tiny domain with perfect tabular data — classical models may be cheaper and auditable.
Strict freshness on facts without retrieval — weights are a frozen snapshot of training data.

How to read the diagrams

The autoregressive loop shows tokenize → transformer → softmax → sample → append context; the attention block zooms into how context steers the next-token distribution. The ascii layer stack below is the same forward pass.

Simple Fundamental Explanation

Imagine you have a highly advanced autocomplete on your phone. If you type “I want to eat a”, your phone suggests “pizza”, “burger”, or “sandwich”.

It doesn’t know what a sandwich is. It doesn’t understand hunger. It has simply read billions of text messages and calculated that, mathematically, the word “pizza” frequently follows the sequence of words “I want to eat a”.

A Large Language Model (LLM) like GPT-4 is essentially this autocomplete concept scaled up to a massive, incomprehensible size. Instead of hardcoding rules of logic, grammar, or facts, an LLM is a giant probabilistic engine. It looks at all the text you’ve provided (the context) and calculates the probability distribution of what the very next piece of a word (a token) should be.

It rolls a weighted die to pick the next token, adds that token to the context, and repeats the process over and over. By predicting one probabilistic token at a time, it generates poetry, writes code, and holds complex conversations.

Deep Dive: How It Works Under the Hood

LLMs are built on the Transformer architecture.

1. Tokenization

Text is chopped into chunks called tokens (words or sub-words). “Unbelievable” might become ["Un", "believ", "able"].

2. Embeddings

Every token is converted into a list of numbers (a vector). This vector represents the “meaning” of the token in a high-dimensional space. Words with similar meanings (Dog, Cat) have vectors pointing in similar directions.

3. The Attention Mechanism

This is the core of the Transformer. When predicting the next token, the model looks back at all previous tokens and decides which ones are important. If the sentence is “The bank of the river is muddy”, the model pays “attention” to “river” and “muddy” to understand that “bank” means land, not a financial institution.

4. Feed-Forward Neural Networks

The contextualized data is passed through massive layers of matrix multiplications (the “weights” of the model). These weights contain compressed, probabilistic representations of all the data the model was trained on.

5. Softmax Output

The final layer outputs a giant list of scores (logits), one for every possible token in the vocabulary (e.g., 50,000 tokens). These scores are converted into probabilities summing to 100%. The model samples from this distribution to output the next token.

Visual Diagram

Diagram 1 — Autoregressive token loop

flowchart LR
  Ctx["Context tokens"]
  Tok["Tokenize + embed"]
  Tr["Transformer blocks"]
  Log["Logits over vocab"]
  Smp["Softmax + sample<br/>temperature T"]
  Out["Append token"]
  Ctx --> Tok --> Tr --> Log --> Smp --> Out
  Out -.->|extend context| Ctx

Diagram 2 — Self-attention over context

flowchart TB
  Q["Query: next position"]
  K["Keys: all prior tokens"]
  V["Values: content vectors"]
  Attn["Attention weights<br/>river · muddy disambiguate bank"]
  Out["Contextual hidden state"]
  Q --> Attn
  K --> Attn
  V --> Attn
  Attn --> Out

Input Context: "The capital of France is"

Layer 1: Tokenize
[ "The", " capital", " of", " France", " is" ]

Layer 2: Transformer Blocks (Attention + Matrix Math)
(Processes context, identifies relationships)

Layer 3: Output Logits (Raw scores for 50,000 possible next words)
"Paris": 15.2
"London": 2.1
"a": -1.5

Layer 4: Softmax (Converts to Probabilities)
"Paris": 98.5%
"London": 0.5%
...

Output: Model samples from distribution and selects " Paris".
New Context: "The capital of France is Paris" -> Repeat.

Practical Example: Temperature Control in AI

When you use the OpenAI API, you can adjust a parameter called Temperature. This directly manipulates the probabilistic nature of the LLM.

Temperature = 0 (Deterministic): The model always picks the single token with the highest probability. The output is rigid, predictable, and highly factual. Ideal for writing code or answering math questions.
Temperature = 1 (Probabilistic): The model respects the true probability distribution. If “Paris” is 80% and “beautiful” is 20%, it will pick “beautiful” 1 out of 5 times. The output is creative and varied. Ideal for writing stories or brainstorming.

By tweaking how we sample the probabilistic output, the same model acts like a rigid calculator or a creative artist.

The Mathematics: Equations and In-Depth Analysis

1. The Core Objective Function

The fundamental goal of an LLM during training is to maximize the likelihood of a sequence of tokens $W = (w_{1}, w_{2}, ..., w_{T})$ . By the chain rule of probability, the joint probability of the entire sequence is the product of conditional probabilities:

P (w_{1}, w_{2}, ..., w_{T}) = t = 1 \prod T P (w_{t} ∣ w_{1}, w_{2}, ..., w_{t - 1})

The neural network acts as a function approximator parameterized by weights $θ$ to estimate $P_{θ} (w_{t} ∣ context)$ .

2. The Softmax Function

To convert raw neural network outputs (logits $z_{i}$ ) into a valid probability distribution where all probabilities sum to 1, the model uses the Softmax function:

P (y = i) = \frac{e ^{z_{i} / T}}{\sum _{j} e ^{z_{j} / T}}

Where $T$ is the Temperature parameter.

If $T = 1$ , it’s the standard distribution.
As $T \to 0$ , the highest logit dominates, and the probability approaches 100% for the top token (argmax).
As $T \to \infty$ , the distribution becomes completely flat (uniform randomness).

3. Scalability and Matrix Math

Why does this scale so well? Unlike traditional deterministic logic trees (if/else statements), the probability estimation $P (w_{t} ∣ context)$ is computed entirely via dense matrix multiplication: $Y = Softmax (Q K^{T}) V$ (The core Attention equation).

Matrix multiplication is highly parallelizable. Modern GPUs (like Nvidia H100s) are designed with thousands of cores specifically built to execute massive matrix multiplications probabilistically across thousands of chips simultaneously. This allows LLMs to scale to trillions of parameters.

Lab

Sample next-token distribution over toy logits as temperature changes.

Temperature TT = 1.0

As T → 0, does the distribution sharpen toward one token?