Tensors and Multilinear Algebra

A scalar is a “rank-0” tensor (a single number). A vector is a “rank-1” tensor (an array). A matrix is a “rank-2” tensor (a grid). Beyond these lie higher-rank tensors—multi-dimensional arrays that follow specific transformation rules. Tensors are the natural language of General Relativity, Quantum Mechanics, and modern Deep Learning.

What is a Tensor?

While computer scientists often define a tensor as “a multidimensional array,” mathematicians define it by how it transforms or what it does.

A tensor is a multilinear map. Just as a matrix $A$ represents a linear map $f(v)$, a tensor $T$ takes multiple vectors as input and produces a scalar (or another vector). For example, a rank-2 tensor $T(u, v)$ is a function that is linear in $u$ AND linear in $v$.

Covariance and Contravariance

In physics, we distinguish between:

• Contravariant vectors ($v^i$): Things like velocity or position that scale “with” the coordinate system.
• Covariant vectors ($w_i$): Things like gradients or dual vectors that scale “against” the coordinate changes.

A general tensor $T^i_j$ can have both types of indices. This distinction is crucial for ensuring that physical laws remain the same regardless of the units or axes we choose.

The Tensor Product

The tensor product $\otimes$ is a way to combine two vector spaces $V$ and $W$ into a larger space $V \otimes W$. If $v \in V$ and $w \in W$, then $v \otimes w$ is an element of the product space. If $V = \mathbb{R}^m$ and $W = \mathbb{R}^n$, the tensor product $V \otimes W$ has dimension $m \times n$. You can think of this as the space of all possible $m \times n$ matrices.

import numpy as np

# Create two vectors
v = np.array([1, 2, 3])
w = np.array([4, 5])

# Compute the Outer Product (a simple tensor product)
# This results in a Rank-2 tensor (Matrix)
T = np.outer(v, w)

print("Vector v (Rank 1):", v)
print("Vector w (Rank 1):", w)
print("\nTensor Product v ⊗ w (Rank 2 Matrix):\n", T)

# Higher-order tensor via expansion
# A Rank-3 tensor could be representing interactions 
# between three spaces.
T3 = np.zeros((3, 2, 2))
T3[0, :, :] = np.eye(2)
T3[1, :, :] = np.ones((2, 2))
T3[2, :, :] = np.zeros((2, 2))

print("\nRank-3 Tensor (3x2x2):\n", T3)
print("\nAccessing a slice T3[0]:\n", T3[0])

Multilinear Forms and Determinants

The determinant is the most famous example of a multilinear form. It is an alternating $(n,0)$-tensor. “Alternating” means that if you swap two input vectors, the sign of the output flips. This property is what allows the determinant to measure signed volume.

Einstein Summation Notation

In tensor calculus, we often drop the summation sign $\sum$. If an index appears twice (once up, once down), it is summed over. Instead of: $y^i = \sum_j A^i_j x^j$ We write: $y^i = A^i_j x^j$ This compact notation is the standard in engineering and theoretical physics.

Exercises