Series: Linear Algebra for Programmers

Series: Deep dives into specific topics

This series starts from 2D rotation matrices and takes us through the developments we have made to understand and describe rotations - it covers imaginary numbers, intrinsic and extrinsic rotations, Euler angles, quaternions, geometric algebra and Lie theory.

Matrix calculus and autodiff

Linear operator

Gradient of a function

Jacobian of a function

Key matrix derivatives

Function is a DAG

Autodiff on a DAG

VJP and JVP

Hessian of a function

Starting from the idea of a small change in output given a small change in input, this series builds the concepts of linear operators, gradients and Jacobians, and covers the foundational ideas behind autodiff alongwith links to implementations in PyTorch and JAX.

From Permutations to Sinkhorn

Overview and permutation cycles

Permutation spectra

Circulant matrices

Birkhoff polytope

Doubly stochastic matrices

Birkhoff-von-Neumann algorithm

Matrix scaling

Sinkhorn distance

Explore the journey from the spectral properties of permutations to the geometry of doubly stochastic matrices, Birkhoff’s polytope, and how it all leads to Sinkhorn distances in optimization and machine learning.