In Part 3, we explored Recurrent Neural Networks (RNNs) to provide sequence memory through hidden states. However, sequential recurrence suffers from performance bottlenecks and memory decay over extended timelines. Modern frontier AI models resolve this by implementing the Transformer Architecture driven by the Self-Attention Mechanism.
Self-Attention evaluates an entire sequence simultaneously, calculating mathematically how every word relates to every other word regardless of positional distance. In this guide, we build the core math and code of Scaled Dot-Product Attention and a Mini Transformer block from scratch using pure Python and NumPy.
1. Queries, Keys, and Values
The attention layer projects input vectors into three distinct spaces:
- Query (Q): Represents the current search target of a token.
- Key (K): Advertises the identity and features of each token.
- Value (V): The actual semantic information extracted when a match occurs.
The core Scaled Dot-Product formula is defined as:
Attention(Q, K, V) = Softmax( (Q · KT) / √dk ) · V
2. Complete Python Script: Scratch Transformer Engine
Below is the complete self-contained script written in pure NumPy:
import numpy as np
class MiniTransformerBlock:
def __init__(self, embed_dim=16, num_heads=2, hidden_dim=32):
assert embed_dim % num_heads == 0, "embed_dim must be divisible by num_heads"
self.embed_dim = embed_dim
self.num_heads = num_heads
self.head_dim = embed_dim // num_heads
self.hidden_dim = hidden_dim
np.random.seed(42)
self.W_q = np.random.randn(embed_dim, embed_dim) * 0.1
self.W_k = np.random.randn(embed_dim, embed_dim) * 0.1
self.W_v = np.random.randn(embed_dim, embed_dim) * 0.1
self.W_o = np.random.randn(embed_dim, embed_dim) * 0.1
self.W_ffn1 = np.random.randn(embed_dim, hidden_dim) * 0.1
self.b_ffn1 = np.zeros((1, hidden_dim))
self.W_ffn2 = np.random.randn(hidden_dim, embed_dim) * 0.1
self.b_ffn2 = np.zeros((1, embed_dim))
def softmax(self, x, axis=-1):
e_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
return e_x / np.sum(e_x, axis=axis, keepdims=True)
def relu(self, x):
return np.maximum(0, x)
def scaled_dot_product_attention(self, Q, K, V, mask=None):
scores = np.matmul(Q, K.swapaxes(-2, -1))
scaled_scores = scores / np.sqrt(self.head_dim)
if mask is not None:
scaled_scores = np.where(mask == 0, -1e9, scaled_scores)
weights = self.softmax(scaled_scores, axis=-1)
return np.matmul(weights, V), weights
def multi_head_attention(self, X):
seq_len = X.shape[0]
Q = np.dot(X, self.W_q)
K = np.dot(X, self.W_k)
V = np.dot(X, self.W_v)
Q_h = Q.reshape(seq_len, self.num_heads, self.head_dim).swapaxes(0, 1)
K_h = K.reshape(seq_len, self.num_heads, self.head_dim).swapaxes(0, 1)
V_h = V.reshape(seq_len, self.num_heads, self.head_dim).swapaxes(0, 1)
attn_out, weights = self.scaled_dot_product_attention(Q_h, K_h, V_h)
concat = attn_out.swapaxes(0, 1).reshape(seq_len, self.embed_dim)
return np.dot(concat, self.W_o), weights
def forward(self, X):
attn_out, weights = self.multi_head_attention(X)
x1 = X + attn_out
ffn_h = self.relu(np.dot(x1, self.W_ffn1) + self.b_ffn1)
x2 = x1 + np.dot(ffn_h, self.W_ffn2) + self.b_ffn2
return x2, weights
if __name__ == "__main__":
tokens = ["AI", "builds", "neural", "models"]
seq_len = len(tokens)
embed_dim = 16
np.random.seed(101)
sample_vectors = np.random.randn(seq_len, embed_dim)
transformer = MiniTransformerBlock(embed_dim=16, num_heads=2, hidden_dim=32)
output_matrix, attention_map = transformer.forward(sample_vectors)
print("Forward pass completed. Output Shape:", output_matrix.shape)
3. Architectural Summary
- Parallel Evaluation: Unlike recurrent units that loop across tokens step-by-step, attention evaluates all pairwise relationships in one matrix operation.
- Global Context: The distance between tokens does not decay context memory, making Transformers ideal for deep natural language tasks.