Master Guide: Sequence Models to Transformers

1. Introduction to Sequence Models

Why Sequences Matter

Traditional neural networks assume that all inputs are independent of each other. However, for many real-world tasks, this is a fatal flaw. Sequence Models are designed specifically for data where the order of the data dictates its meaning.

Speech Recognition: Audio clips mapped to text transcripts over time.
Music Generation: Predicting the next note in a continuous melody.
Machine Translation: Translating English to French based on sentence grammar.
DNA Sequence Analysis: Understanding biological chains.

Interactive: The Forest Hiker

Toggle the button below to see how adding historical context (a sequence) changes wild guessing into highly accurate predictions.

Prediction Status

Without past data, the person could walk in any of 360 degrees. Probability is uniform (wild guessing).

2. Problem Formulations for Sequence Models

Sequence models are highly flexible. Depending on the task, we can architect the network to change how the input sequence ($X$) and output sequence ($Y$) are processed across time steps. Select a formulation below to learn how it works.

Many-to-One (Sentiment)

Applied Mathematical Trace

Awaiting forward pass...

4. Building Recurrent Neural Networks (RNNs)

In Simple Words: The Problem with Amnesia

Imagine reading a book, but after every single word, your memory is completely wiped. You read "The", then forget it. You read "movie", then forget it. You read "is", then forget it. By the time you read "great", you have absolutely no idea what is great. This is exactly how a Standard Feedforward Network works. It processes each piece of data in total isolation, completely ignoring the element of time.

The Solution: A Memory Pad (The Hidden State)

To fix this, we give the network a scratchpad. When it reads the first word, it writes down some notes. When it moves to the second word, it doesn't just read the word—it reads the word AND the notes from the previous step. It combines them, makes a new prediction, and updates the notes for the next step.

This running memory pad is called the Hidden State ($h_t$). Passing this memory forward is what creates the Recurrence Relation, allowing the AI to understand sentences, time-series, and context.

Interactive: Unrolling the Timeline

				        Timeline: Waiting for you to start...
				        3 Seconds per Step
				    

Standard Network: Awaiting start command...

Recurrent Network (RNN): Awaiting start command...

4. Formalizing RNN Internals & Architecture

The Mathematics of the Hidden State

An RNN processes sequences by applying the exact same recurrence relation at every step.

$$h_t = f_W(h_{t-1}, x_t)$$

1. Update Hidden State: Multiply inputs and past memory by their shared weight matrices, add biases, and squash with tanh.

$$h_t = \tanh(W_{hx} \cdot x_t + W_{hh} \cdot h_{t-1} + b_h)$$

2. Compute Output: Use the newly updated hidden state to make a prediction.

$$y_t = W_{yh} \cdot h_t + b_y$$

TensorFlow Pseudo-Code

Notice how the weight variables are initialized once, but called in a loop.

rnn_architecture.py

				class SimpleRNNCell:
				    def __init__(self, input_size, hidden_size, vocab_size):
				        # Define Shared Weight Matrices once
				        self.W_hx = tf.Variable(tf.random.normal([input_size, hidden_size]))
				        self.W_hh = tf.Variable(tf.random.normal([hidden_size, hidden_size]))
				        self.W_yh = tf.Variable(tf.random.normal([hidden_size, vocab_size]))
				        self.b_h  = tf.Variable(tf.zeros([hidden_size]))
				        self.b_y  = tf.Variable(tf.zeros([vocab_size]))
				
				    def call(self, x_t, h_prev):
				        # 1. Update State
				        h_t = tf.nn.tanh(tf.matmul(x_t, self.W_hx) + 
				                         tf.matmul(h_prev, self.W_hh) + self.b_h)
				                         
				        # 2. Output Next Word
				        y_t = tf.matmul(h_t, self.W_yh) + self.b_y
				        
				        return h_t, y_t
				
				# Processing the sequence over time
				h_t = tf.zeros([1, hidden_size]) # Initialize memory to zero
				for x_t in sequence:
				    h_t, y_t = rnn_cell.call(x_t, h_t)

Computational Graph: Rolled Network

● Shared Parameters

Graph Architecture Status

Rolled State: The traditional way to draw an RNN, showing the recurrence relation looping back into itself.

5. Training RNNs (Computing Sequence Loss)

The Mathematics of Sequence Loss

Unlike standard networks that calculate one final error, an RNN generating a sequence calculates an error at every single time step.

1. Individual Loss Slice: At time $t$, the model outputs prediction $\hat{y}_t$. We compare it to the ground truth target $y_t$.

$$L_t = \text{Loss}(\hat{y}_t, y_t)$$

2. Total Sequence Loss: We sum the individual losses across the entire sequence to get the final score. This single number is what Backpropagation uses to update the weights.

$$L = \sum L_t$$

Operationalizing in TensorFlow

Setting return_sequences=True ensures the layer outputs $\hat{y}_t$ at every step instead of just the final state.

train_rnn.py

				import tensorflow as tf
				
				# 1. Define the layers
				rnn_layer = tf.keras.layers.SimpleRNN(hidden_size, return_sequences=True)
				dense_layer = tf.keras.layers.Dense(vocab_size)
				
				# 2. Forward pass for the WHOLE sequence
				hidden_states = rnn_layer(inputs) 
				predictions = dense_layer(hidden_states)
				
				# 3. Compute Total Sequence Loss automatically
				loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
				total_loss = loss_fn(targets, predictions)

Interactive Engine: Loss Computation Graph

● Next-Word Prediction Task

LOSS CALCULATION TRACE

Awaiting forward pass...

6. Real-World Sequence Modeling & Embeddings

Neural networks cannot do math on letters. Learn the design criteria for sequences and how we convert human words into dense numerical vectors.

The Embedding Pipeline

● Task: Next Word Prediction

Type a sentence below to see how a model extracts the vocabulary, indexes it, and generates a dense vector embedding for the neural network to process.

7. Backpropagation Through Time (BPTT)

RNNs are trained by backpropagating errors across individual time steps. This mathematically leads to the practical challenges of "exploding" and "vanishing" gradients, making it difficult for standard RNNs to track long-term dependencies.

Recall: Backpropagation in Feed Forward Models

The standard Backpropagation algorithm:

Take the derivative (gradient) of the loss with respect to each parameter.
Shift parameters in order to minimize loss.

RNNs: Backpropagation Through Time

					            ---> Forward Pass (Data flows left-to-right)

					            <--- Backward Pass (Errors flow right-to-left)

Standard RNN Gradient Flow

Because the network is unrolled across time, the gradient must travel backward through the entire sequence.
Computing the gradient with respect to h₀ involves many factors of W_hh + repeated gradient computation!

Standard RNN Gradient Flow Issues

1. Exploding Gradients

Cause: Many values > 1. Repeated multiplication causes the gradient to grow exponentially.
Solution: Gradient clipping to scale big gradients down to a safe threshold.

2. Vanishing Gradients

Computing the gradient wrt h₀ involves many factors of W_hh. If these values are small, multiplying them repeatedly shrinks the gradient to zero.

Many values < 1 lead to vanishing gradients. Root causes include:

Activation function: Derivatives of functions like tanh are strictly ≤ 1.
Weight initialization: Starting weights too small.
Network architecture: The deep sequential nature of standard RNNs.

The Problem of Long-Term Dependencies

Why are vanishing gradients a problem?

You multiply many small numbers together during the backward pass.
Errors due to further back time steps have smaller and smaller gradients.
Because early steps get no gradient signal, the network biases parameters to capture only short-term dependencies, completely forgetting distant context.

BPTT Simulator: Gradient Clipping

● 3-Phase Animation

BPTT COMPUTATION TRACE

Waiting for input...

8. LSTMs and Gating Mechanisms

Solving Gradient Instability

To solve the gradient instability that destroys standard RNNs, Long Short-Term Memory (LSTM) networks are introduced. Instead of overwriting memory at every step, an LSTM selectively controls information flow.

Gating Mechanisms in Neurons

The core idea is to use gates to selectively add or remove information within each recurrent unit. A gate uses a Sigmoid neural network layer to output numbers between 0 and 1, describing how much of a component should be let through (0 = block completely, 1 = pass everything).

Modern sequence architectures heavily rely on this, including the gated cell LSTM and the GRU (Gated Recurrent Unit).
LSTMs rely on a gated cell to track information throughout many time steps without the gradients vanishing.

Interactive LSTM Cell Architecture

● Click gates to explore

The LSTM Cell State

Select a gate button to the left to see how information selectively flows through the LSTM cell.

Notice the straight line running across the top (C_t). That is the Cell State. It acts like a conveyor belt carrying long-term memory. Because the interactions with it are strictly linear (just simple multiplication and addition), gradients can flow backward through it without vanishing!

9. Limitations of Recurrent Models

Evaluate the architectures against the desired capabilities: Continuous Output, Parallelization, and Long Memory.

Architecture Evaluator

● Can we eliminate recurrence?

Desired Capabilities

Continuous Stream:

Parallelization:

Long Memory:

10. Introduction to Attention & Transformers

Eliminating Recurrence Entirely

To overcome RNN limitations, researchers asked: What if we process the entire sequence all at once?

This completely eliminates the sequential bottleneck, allowing for massive parallelization on modern GPUs. However, without a step-by-step memory loop, the network needs a new way to understand context. This introduces the "Attention" mechanism and Transformer networks, which process inputs simultaneously by mathematically identifying and attending to the most important parts of a sequence.

The Intuition Behind Self-Attention

Self-Attention solves the context problem by treating sequence comprehension as a search problem. For every single word in a sequence, the model searches the rest of the sentence to figure out what other words matter to it.

1. Identify which parts to attend to: The model compares a word to every other word, assigning a high "attention score" to highly relevant words (e.g., a pronoun and the noun it refers to).
2. Extract the features: The model then pulls the mathematical meaning from those highly-scored words and blends them into the original word, enriching its context.

Interactive Self-Attention Visualizer

● Hover over any word to view its "Search" results

Attention Analysis

Hover over a word above to see how it "attends" to the rest of the sequence to build context.

11. The Attention Mechanism (Query, Key, Value)

The Database Search Analogy

To mathematically execute the "search problem" introduced in the last topic, Attention uses an intuitive concept borrowed directly from database retrieval and search engines:

Think about how you search for a video on YouTube:

Query (Q): What you type into the search bar (e.g., "funny cat"). In Attention, this is the current word trying to figure out its context.
Key (K): What the database holds (e.g., the title, tags, and description of every video). In Attention, every other word in the sentence generates a "Key" describing what it is.
Value (V): The actual video you watch. In Attention, this is the mathematical meaning of the word that will be extracted and blended into the current word.

The 2-Step Extraction Process

Every single word in a sequence generates its own Query, Key, and Value vectors. The network then runs a two-step process:

1. Compute the Attention Mask:

$$Q \cdot K$$

How similar is the Key to the Query? The network computes a dot product between the Query vector of the current word and the Key vectors of all surrounding words. High similarity results in a high percentage score.

2. Extract Values based on Attention:

$$\text{Score} \times V$$

Return the values with the highest attention. The network multiplies those percentage scores by the actual Value vectors of the words. It sums them all up to create a brand new, highly contextualized representation of the original word.

Interactive Q, K, V Visualizer

● Select a scenario to see how the math adapts

12. Self-Attention Operations

Goal: Identify and attend to the most important features in the input.
Data is fed in all at once! The network uses linear algebra to build context dynamically.

[Image of Multi-Head Attention mechanism block diagram]

1. Encode Position Information

Because there is no recurrence, we add a positional embedding to the standard word embedding to stamp each word with its sequence order.

2. Extract Query, Key, Value

The Pos-Aware Embedding is multiplied by three separate Linear Layers (learned weights) to generate our Q, K, and V search matrices.

3. Compute Attention Weighting

Compute pairwise similarity using a Dot Product ($Q \cdot K^T$). Scale it, and apply Softmax to map similarities to percentages.

4. Extract Features

Multiply the final Attention Weights matrix by the Value matrix. This creates a contextualized Output vector for each word.

The Attention Matrix "Chessboard"

● Visualizing $\text{Softmax}(Q \cdot K^T)$

Keys (K) - "What the sequence holds"

Queries (Q) - "Searching"

13. Transformer Architecture & Applications

See how Multi-Head Attention applies to Computer Vision, allowing the AI to pull apart complex visual scenes feature by feature.

Stacking Multiple Attention Heads

A single self-attention head extracts one specific type of relationship. But what if the input is highly complex, like a full sentence or an entire photograph? To increase the capacity of the network, we stack Multiple Attention Heads in parallel.

Each head has its own set of Q, K, and V weights, meaning each head learns to "look" at something completely different in the exact same input.

Head 1 might attend to Living Entities (Humans, Animals).
Head 2 might attend to Infrastructure (Buildings, Roads).
Head 3 might attend to Nature (Trees, Weather).

Real-World Applications

Attention isn't just for words. It is a universal feature extractor. This architecture revolutionized multiple fields:

Natural Language Processing (LLMs): Models like GPT and Gemini process thousands of words simultaneously, with heads tracking grammar, tone, and logic.
Biological Sequences: Models like AlphaFold use attention to track physical relationships between amino acids to predict 3D protein structures.
Computer Vision (ViT): Instead of words, an image is chopped into patches. The Transformer attends to these patches, pulling out structural, biological, and environmental features exactly like the interactive example below!

Vision Transformer: Multi-Head Attention Visualizer

Awaiting Attention...

Select an Attention Head above to see how the Vision Transformer isolates specific features.

[Living Data]

[Infra Data]

[Nature Data]

→ Feed Forward Network