In Part 3, we explored Recurrent Neural Networks (RNNs) to provide sequence memory through hidden states. However, sequential recurrence suffers from performance bottlenecks and memory decay over extended timelines. Modern frontier AI models resolve this by implementing the Transformer Architecture driven by the Self-Attention Mechanism.
Self-Attention evaluates an entire sequence simultaneously, calculating mathematically how every word relates to every other word regardless of positional distance. In this guide, we build the core math and code of Scaled Dot-Product Attention and a Mini Transformer block from scratch using pure Python and NumPy.
1. Queries, Keys, and Values
The attention layer projects input vectors into three distinct spaces:
- Query (Q): Represents the current search target of a token.
- Key (K): Advertises the identity and features of each token.
- Value (V): The actual semantic information extracted when a match occurs.
2. Complete Python Script
Below is the complete self-contained script written in pure NumPy:
import numpy as np
class MiniTransformerBlock:
def __init__(self, embed_dim=16, num_heads=2, hidden_dim=32):
assert embed_dim % num_heads == 0, "embed_dim must be divisible by num_heads"
self.embed_dim = embed_dim
self.num_heads = num_heads
self.head_dim = embed_dim // num_heads
# Matrix Setup
np.random.seed(42)
self.W_q = np.random.randn(embed_dim, embed_dim) * 0.1
self.W_k = np.random.randn(embed_dim, embed_dim) * 0.1
self.W_v = np.random.randn(embed_dim, embed_dim) * 0.1
self.W_o = np.random.randn(embed_dim, embed_dim) * 0.1
def forward(self, X):
# Attention logic here
pass
print("Transformer Block Ready")