In Part 4, we built a Transformer block capable of Self-Attention. However, if we want our AI to generate text like GPT (Generative Pre-trained Transformer), it must learn to predict the future without "cheating" by looking ahead. This requires a structural modification known as Causal Masking.
During training, we feed entire sentences into the Transformer simultaneously for speed. To prevent a token from reading the words that come after it, we apply a mathematical mask. In this guide, we will implement Causal Masking and build the autoregressive loop that allows our AI engine to generate continuous text.
1. The Logic of Causal Masking
A causal mask is a lower-triangular matrix filled with zeros and negative infinities. Before the Softmax function calculates attention probabilities, we apply this mask to the raw attention scores.
- Allowed Context: A token can attend to itself and all previous tokens (values remain unchanged).
- Blocked Context: Future tokens are masked with -infinity. When passed through Softmax, e-infinity becomes exactly 0, completely erasing the future from the network's vision.
2. The Autoregressive Generation Loop
Once trained, the model generates text autoregressively (step-by-step):
- The user provides a starting prompt.
- The network calculates the probability distribution for the next token.
- The highest probability token is selected and appended to the input sequence.
- The entire new sequence is fed back into the network to predict the next word.
3. Complete Python Script: Generative Transformer Loop
Below is the Python implementation integrating the causal mask and a text generation pipeline using NumPy:
import numpy as np
def create_causal_mask(seq_len):
# Creates a matrix where upper triangle is 1 (to be masked) and lower is 0
mask = np.triu(np.ones((seq_len, seq_len)), k=1)
return mask
def apply_mask(attention_scores, mask):
# Replace 1s in the mask with -infinity
masked_scores = np.where(mask == 1, -1e9, attention_scores)
return masked_scores
def softmax(x, axis=-1):
e_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
return e_x / np.sum(e_x, axis=axis, keepdims=True)
# Simulated Generator Function
def generate_text(prompt_tokens, max_length=5, vocab_size=100):
print(f"Initial Prompt: {prompt_tokens}")
current_sequence = prompt_tokens.copy()
for step in range(max_length):
seq_len = len(current_sequence)
# 1. Create Causal Mask for current sequence length
causal_mask = create_causal_mask(seq_len)
# 2. Simulate Attention Scores (Randomized for this example)
raw_scores = np.random.randn(seq_len, seq_len)
# 3. Apply Mask & Softmax
masked_scores = apply_mask(raw_scores, causal_mask)
attention_weights = softmax(masked_scores, axis=-1)
# 4. Predict Next Token (Simulated via random selection weighted by last token state)
next_token_id = np.random.randint(0, vocab_size)
current_sequence.append(next_token_id)
print(f"Step {step+1} | Generated Token ID: {next_token_id}")
return current_sequence
if __name__ == "__main__":
# Test Causal Mask
print("--- 4x4 Causal Mask Matrix ---")
print(create_causal_mask(4))
print("\n--- Autoregressive Generation Loop ---")
final_output = generate_text(prompt_tokens=[12, 45], max_length=4)
print(f"Final Sequence: {final_output}")
4. Architectural Summary
By forcing the attention matrix to ignore future inputs, we ensure the Transformer learns actual sequential causality rather than just memorizing static datasets. This exact masking logic powers the core reasoning engine of every modern Large Language Model.