Large Language Models (LLMs): Architecture, Tokenization & Transformer Decoders

Over the past few years, AI systems such as ChatGPT, Claude, Gemini, and Llama have redefined human-computer interaction. They write software, translate nuanced languages, summarize dense technical literature, answer complex conceptual queries, and generate creative literature. At the technological epicenter of this revolution lies the Large Language Model (LLM).

At its mathematical essence, an LLM is a machine learning model trained on hundreds of billions or trillions of tokens of text data to discover statistical patterns of human thought and communication. Its fundamental operational mechanism is deceptively straightforward: given an input sequence of tokens, predict the probability distribution over the entire vocabulary for the immediate next token.

A raw pretrained model given the prompt "The capital of France is" may complete it with " Paris". However, if given "Write an essay on photosynthesis:", an unaligned base model might simply generate more multiple-choice questions about photosynthesis because it learned from exam papers. To transform a raw statistical sequence predictor into a reliable assistant, modern engineering uses post-training alignment, chat templates, and system prompt framing.

How does predicting one word at a time generate cohesive essays and working code? The theoretical foundation of language modeling is the probabilistic chain rule of conditional probabilities.

When generating text, the LLM operates in an autoregressive loop:

1. Step 1: Feed prompt tokens $\mathbf{x} = (x_1, \dots, x_t)$ into the Transformer.
2. Step 2: Compute logits for position $t$ across all tokens in vocabulary $V$.
3. Step 3: Convert logits to probabilities using Softmax (scaled by temperature).
4. Step 4: Select candidate token $x_{t+1}$ using a decoding policy (e.g., Top-$p$ sampling).
5. Step 5: Append $x_{t+1}$ to the input sequence: $\mathbf{x}_{\text{new}} = (x_1, \dots, x_t, x_{t+1})$.
6. Step 6: Repeat until the model emits the designated End-of-Sequence token (e.g., <|endoftext|> or <|im_end|>) or reaches the max token budget.

Neural networks cannot process raw UTF-8 string characters directly. They perform floating-point tensor operations. Tokenization is the deterministic algorithm that fragments human text into discrete atomic chunks called tokens and maps each token to an integer identifier.

Byte-Pair Encoding (BPE) Algorithm

Invented by Philip Gage in 1994 as a data compression format and adapted for NLP by Sennrich et al. (2016), BPE starts with individual bytes as base vocabulary and iteratively merges the most frequent adjacent pairs:

After tokenization, discrete token integers $\mathbf{x} \in \mathbb{N}^S$ enter the model. The first layer is a learned Token Embedding Matrix $\mathbf{W}_e \in \mathbb{R}^{V \times D}$, where $V$ is vocabulary size and $D$ is model hidden dimension ($D = 4096$ in Llama 3 8B).

A token ID $k$ acts as a row index lookup into $\mathbf{W}_e$: $$\mathbf{e}_k = \mathbf{W}_e[k, :] \in \mathbb{R}^D$$

Why Self-Attention Is Permutation Invariant

The core attention equation $\text{Softmax}(Q K^\top / \sqrt{d_k}) V$ computes pairwise dot products. If you shuffle the token order, the dot products yield the identical set of values. Without explicit positional information, the network cannot distinguish between "The dog bit the man" and "The man bit the dog".

Unlike the original 2017 "Attention Is All You Need" encoder-decoder paper designed for machine translation, modern autoregressive LLMs utilize a Decoder-Only Pre-Norm Transformer architecture.

The Causal Mask Matrix $\mathbf{M}$

In masked self-attention, we enforce causality by adding a lower-triangular additive mask matrix $\mathbf{M} \in \mathbb{R}^{S \times S}$:

Because $\exp(-\infty) = 0$, applying the Softmax operator forces all attention weights for future positions to identically zero: $$\text{Attention Weight}_{ij} = \frac{\exp\left(\frac{\mathbf{q}_i^\top \mathbf{k}_j}{\sqrt{d_k}} + M_{ij}\right)}{\sum_{l=1}^S \exp\left(\frac{\mathbf{q}_i^\top \mathbf{k}_l}{\sqrt{d_k}} + M_{il}\right)} = 0 \quad \forall j > i$$ This mathematical property guarantees that token $i$ can never see or attend to token $i+1$, preserving autoregressive integrity.

RMSNorm vs LayerNorm

Classical LayerNorm computes mean $\mu$ and variance $\sigma^2$: $$\text{LN}(x) = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \odot \gamma + \beta$$ Zhang and Sennrich (2019) demonstrated that the mean-centering operation $\mu$ adds unnecessary computation without improving generalization. RMSNorm (Root Mean Square Normalization) normalizes purely by root mean square: $$\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \epsilon}} \odot \gamma$$ This delivers an estimated $7\%$ to $12\%$ speedup across deep layers while preserving complete numerical stability.

Not all Transformer models are structured alike. The deep learning literature features three foundational archetypes:

Pretraining is the most computationally demanding phase of LLM creation, typically consuming millions of GPU-hours on clusters of H100s. The training pipeline processes trillions of tokens scraped from web documents (Common Crawl), curated books, high-quality codebases (GitHub), scientific arXiv papers, and encyclopedic knowledge.

Pretraining does not memorize facts verbatim; it compresses statistical dependencies, syntactic grammars, world models, and semantic logic into weight matrices $\theta$.

Once the final decoder block outputs hidden vectors $\mathbf{h}_S \in \mathbb{R}^D$, the Language Model Head projects them to the vocabulary dimension: $\mathbf{z} = \mathbf{h}_S \mathbf{W}_{\text{head}} \in \mathbb{R}^V$. How we select the next token from logit vector $\mathbf{z}$ governs the entire personality, creativity, and precision of the LLM.

Temperature Scaling

Temperature $T > 0$ modifies the logit scale before computing Softmax: $$P_i = \frac{\exp(z_i / T)}{\sum_{j=1}^V \exp(z_j / T)}$$

• Low Temperature ($T \to 0$, e.g. $T = 0.2$): Sharpens the distribution. The highest logit dominates; behaves almost like greedy decoding. Ideal for coding and math.
• Standard Temperature ($T = 0.7 - 1.0$): Balanced natural human-like cadence.
• High Temperature ($T > 1.2$, e.g. $T = 1.8$): Flattens the distribution towards uniform entropy. Encourages novelty but risks hallucinations and syntactic collapse.

Top-$k$ Sampling

Truncates the candidate pool to only the top $k$ tokens with the highest logits, setting all other logits to $-\infty$. For example, with $k=50$, only the 50 most probable tokens can ever be sampled.

Top-$p$ (Nucleus) Sampling (Holtzman et al., 2020)

Instead of a fixed count $k$, Top-$p$ dynamically adjusts the candidate pool size based on confidence. It selects the smallest set of tokens $V^{(p)}$ whose cumulative probability exceeds threshold $p \in (0, 1]$: $$\sum_{i \in V^{(p)}} P_i \ge p$$ When the model is confident (one token has $95\%$ probability), the candidate set shrinks to $1$ token. When the context is ambiguous, the candidate set automatically expands to include dozens of plausible alternatives.

In naive autoregressive generation, generating token $t+1$ requires passing all tokens $1 \dots t$ through the network. As sequence length $S$ reaches thousands of tokens, recomputing past Key and Value vectors yields a catastrophic $\mathcal{O}(S^2)$ computational complexity per generated token.

The Context Window defines the maximum sequence length $S_{\max}$ that an LLM can ingest in a single forward pass (e.g. 8k for original Llama 2, 128k for Llama 3, 1M+ for Gemini 1.5).

Pretrained models are statistical raw materials. Transforming them into obedient, safe conversational assistants requires a multi-stage post-training pipeline:

Because LLMs are probabilistic token predictors rather than factual knowledge graph query engines, they are subject to fundamental limitations:

The following production-grade script loads an autoregressive causal language model using Hugging Face transformers, tokenizes a prompt, and executes greedy decoding, temperature sampling, and streaming generation.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer

# 1. Device configuration (CUDA GPU if available)
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Executing inference on device: {device}")

# 2. Load Tokenizer and Causal Language Model
model_id = "gpt2"  # Educational compact decoder; replace with "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
model.eval()

# Ensure padding token exists
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# 3. Format input prompt
prompt = "Artificial intelligence and large language models will"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

print(f"Prompt: '{prompt}'")
print(f"Input Token IDs: {inputs['input_ids'].tolist()[0]}")
print(f"Token Count: {inputs['input_ids'].shape[1]}")

# 4. Strategy A: Deterministic Greedy Decoding
with torch.no_grad():
    greedy_output = model.generate(
        **inputs,
        max_new_tokens=30,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id
    )
print("\n[Strategy A: Greedy Decoding Output]")
print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))

# 5. Strategy B: Stochastic Sampling with Temperature, Top-k, and Top-p
with torch.no_grad():
    sample_output = model.generate(
        **inputs,
        max_new_tokens=30,
        do_sample=True,
        temperature=0.8,
        top_k=50,
        top_p=0.92,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.eos_token_id
    )
print("\n[Strategy B: Temperature (0.8) + Top-p (0.92) Sampling Output]")
print(tokenizer.decode(sample_output[0], skip_special_tokens=True))

# 6. Strategy C: Real-Time Token Streaming Generation
print("\n[Strategy C: Real-Time Token Streaming]")
streamer = TextStreamer(tokenizer, skip_prompt=True)
with torch.no_grad():
    _ = model.generate(
        **inputs,
        max_new_tokens=40,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        streamer=streamer,
        pad_token_id=tokenizer.eos_token_id
    )