Understanding Matrix Multiplication from FastAI

NoteRelated Content

This post is part of our deep learning foundations series. You might also be interested in: - Data Science Steps Series - Feature Preprocessing

1 Matrix Multiplication: The Building Block of Deep Learning

Matrix Multiplication Visualization

In this post, we’ll explore matrix multiplication from first principles, following Jeremy Howard’s excellent teaching approach from the FastAI course. We’ll understand why it’s crucial for deep learning and implement it from scratch in Python.

1.1 Why Matrix Multiplication Matters

Matrix multiplication is fundamental to deep learning because:

1. It’s the core operation in neural network layers
2. It enables efficient parallel computation
3. It allows us to represent complex transformations compactly

1.2 Implementation from Scratch

Let’s implement matrix multiplication using Python and NumPy:

import numpy as np
import torch
from typing import List, Tuple
import matplotlib.pyplot as plt

def matmul(a: List[List[float]], b: List[List[float]]) -> List[List[float]]:
    """Matrix multiplication from scratch"""
    # Check dimensions
    assert len(a[0]) == len(b), "Incompatible dimensions"
    
    # Initialize result matrix
    result = [[0.0 for _ in range(len(b[0]))] for _ in range(len(a))]
    
    # Perform multiplication
    for i in range(len(a)):
        for j in range(len(b[0])):
            for k in range(len(b)):
                result[i][j] += a[i][k] * b[k][j]
    
    return result

# Example matrices
A = [[1, 2], [3, 4]]
B = [[5, 6], [7, 8]]

# Calculate result
result = matmul(A, B)
print("Result of matrix multiplication:")
print(np.array(result))

Result of matrix multiplication:
[[19. 22.]
 [43. 50.]]

1.3 Visualizing Matrix Multiplication

Let’s create a visual representation of how matrix multiplication works:

def plot_matrix_mult(A: np.ndarray, B: np.ndarray) -> None:
    """Visualize matrix multiplication process"""
    fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))
    
    # Plot first matrix
    ax1.imshow(A, cmap='viridis')
    ax1.set_title('Matrix A')
    
    # Plot second matrix
    ax2.imshow(B, cmap='viridis')
    ax2.set_title('Matrix B')
    
    # Plot result
    result = np.dot(A, B)
    ax3.imshow(result, cmap='viridis')
    ax3.set_title('A × B')
    
    plt.tight_layout()
    plt.show()

# Create example matrices
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

plot_matrix_mult(A, B)

1.4 PyTorch Implementation

In practice, we use optimized libraries like PyTorch:

# Convert to PyTorch tensors
A_torch = torch.tensor(A, dtype=torch.float32)
B_torch = torch.tensor(B, dtype=torch.float32)

# PyTorch matrix multiplication
result_torch = torch.matmul(A_torch, B_torch)
print("PyTorch result:")
print(result_torch)

PyTorch result:
tensor([[19., 22.],
        [43., 50.]])

1.5 Performance Comparison

Let’s compare our implementation with NumPy and PyTorch:

import time

def benchmark_matmul(size: int = 100) -> None:
    """Compare performance of different implementations"""
    # Generate random matrices
    A = np.random.randn(size, size)
    B = np.random.randn(size, size)
    
    # Custom implementation
    start = time.time()
    _ = matmul(A.tolist(), B.tolist())
    custom_time = time.time() - start
    
    # NumPy
    start = time.time()
    _ = np.dot(A, B)
    numpy_time = time.time() - start
    
    # PyTorch
    A_torch = torch.tensor(A)
    B_torch = torch.tensor(B)
    start = time.time()
    _ = torch.matmul(A_torch, B_torch)
    torch_time = time.time() - start
    
    print(f"Custom implementation: {custom_time:.4f}s")
    print(f"NumPy: {numpy_time:.4f}s")
    print(f"PyTorch: {torch_time:.4f}s")

benchmark_matmul()

Custom implementation: 0.0618s
NumPy: 0.0094s
PyTorch: 0.0001s

1.6 Key Takeaways

1. Matrix multiplication is a fundamental operation in deep learning
2. Understanding it from first principles helps debug neural networks
3. Libraries like PyTorch provide highly optimized implementations
4. The operation is inherently parallelizable

TipFastAI Insight

Jeremy Howard emphasizes understanding matrix multiplication from scratch because it’s the foundation of neural network operations. This understanding helps in debugging and optimizing deep learning models.

1.7 Next Steps

In future posts, we’ll explore: - How matrix multiplication enables neural network layers - Efficient implementations using CUDA - Common optimization techniques

1.8 Related Posts

• Data Science Steps Series
• Feature Preprocessing
• Using Nougat for Research Papers

1.9 References

1. FastAI Course
2. Deep Learning Book - Linear Algebra Chapter
3. PyTorch Documentation