Back to Articles NLP & LLMs • 18 min read

The Rise of Large Language Models (LLMs) & Transformers

From simple email autocompletion widgets to highly advanced conversational agents capable of debugging code and passing licensing exams, Large Language Models (LLMs) have completely redefined our relationship with computing. This article explores the architecture behind these systems, the mathematical formulation of self-attention, the tokenization pipeline, and how they scale from simple predictors to intelligent reasoning systems.

---

1. The Self-Attention Revolution

Before 2017, NLP tasks relied heavily on Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks. While effective for short sequences, these models processed text sequentially (word-by-word), which meant they struggled to retain long-range context. Furthermore, sequential processing could not be easily parallelized on modern GPU hardware, limiting training dataset scale.

The landmark 2017 research paper "Attention Is All You Need" by Vaswani et al. introduced the **Transformer architecture**. The core innovation of the Transformer is **Self-Attention**—a mechanism that allows the model to process all tokens in a sequence simultaneously and weigh their contextual relationships.

How Attention is Calculated

Self-attention works by mapping each input token representation into three vectors: **Queries (Q)**, **Keys (K)**, and **Values (V)**. The model calculates attention scores by taking the dot product of the Query vector of one word with the Key vector of all other words in the sentence. These scores are scaled, normalized via a Softmax function, and used to compute a weighted sum of the Value vectors:

Where d_k is the dimensionality of the Key vectors. This scaling factor keeps gradients stable during backpropagation.

---

2. Tokenization: Breaking Down the Language

Computers do not understand text directly. When a user inputs a prompt, the system passes the text through a **tokenizer** to convert strings into arrays of numerical indices. The vocabulary is typically constructed using algorithms like **Byte-Pair Encoding (BPE)**.

Tokenization Strategy	Pros	Cons
Character-Level	Small vocabulary size; no out-of-vocabulary words.	Extremely long sequences; difficult to learn long-range semantic patterns.
Word-Level	Intuitive; matches human language structures.	Requires massive vocabularies; cannot handle typos or new words.
Sub-word (BPE/WordPiece)	Balanced sequence length; handles prefixes and suffixes efficiently.	Occasionally creates unintuitive subdivisions.

---

3. Training Pipelines: Pre-training vs Fine-tuning

Modern LLMs are trained in distinct hierarchical phases:

• Pre-training (Causal Language Modeling): The model reads billions of web pages and books. Its objective is to predict the next token given preceding tokens. In doing so, the model builds general syntactic knowledge, common sense, and logic. This phase requires thousands of GPUs and months of compute time.
• Supervised Fine-Tuning (SFT): The model is trained on curated instruction-response datasets. It learns to act as an assistant, answering prompts instead of simply completing text.
• Reinforcement Learning from Human Feedback (RLHF): Humans rate model outputs. A reward model is trained on these ratings, and the LLM is optimized using Proximal Policy Optimization (PPO) to maximize positive feedback while aligning outputs with safety policies.

---

4. Mathematical Softmax Output Selection

At the final layer of the network, output logits are passed through a Softmax function to convert raw scores into a probability distribution over the vocabulary. The model selects the next token using methods like temperature scaling and Top-P filtering:

import numpy as np

def softmax(logits, temperature=1.0):
    # Adjust logits by temperature to control creativity
    logits = np.array(logits) / temperature
    exp_logits = np.exp(logits - np.max(logits))
    return exp_logits / np.sum(exp_logits)

# Simulated vocabulary logits for: ["robot", "human", "code", "learning"]
sample_logits = [2.5, 0.1, 4.2, 1.8]
probabilities = softmax(sample_logits, temperature=0.7)

for word, prob in zip(["robot", "human", "code", "learning"], probabilities):
    print(f"Word: {word:<10} Probability: {prob:.4f}")
            

Through this probability-driven generation, LLMs synthesize ideas, write code, and assist in scientific research across the globe.