Basis and Dimension

The concepts of basis and dimension provide a way to “measure” the size and complexity of a vector space.

Linear Independence and Spanning

A set of vectors $\{\mathbf{v}_1, \dots, \mathbf{v}_n\}$ is linearly independent if the only solution to $c_1\mathbf{v}_1 + \dots + c_n\mathbf{v}_n = \mathbf{0}$ is $c_i = 0$ for all $i$. The span of a set is the set of all possible linear combinations.

import numpy as np

def check_linear_independence(vectors):
  matrix = np.array(vectors).T
  rank = np.linalg.matrix_rank(matrix)
  is_independent = rank == len(vectors)
  return is_independent, rank

# Test linearly independent vectors
v1 = [1, 0, 0]
v2 = [0, 1, 0]
print(f"Independent: {check_linear_independence([v1, v2])}")

# Test linearly dependent vectors
v3 = [1, 1, 0]
print(f"Independent with v3: {check_linear_independence([v1, v2, v3])}")

Definition of a Basis

A basis $B$ for a vector space $V$ is a set of vectors that:

1. Is linearly independent.
2. Spans $V$.

Dimension

The dimension of a vector space $V$, denoted $\dim(V)$, is the number of vectors in any basis for $V$.

import numpy as np

def get_dimension(matrix):
  # The dimension of the column space is the rank
  return np.linalg.matrix_rank(matrix)

A = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
print(f"Rank (Dimension of Col Space) of A: {get_dimension(A)}")
print("A rank of 2 means the vectors are coplanar in R^3.")

Canonical Forms

Sometimes, a matrix is “ugly”—it is filled with dense numbers that obscure the underlying physics or logic of the system. Canonical forms are the “simplest” possible representations of a linear operator. By changing our basis, we can reveal the true nature of the transformation.

Why Canonical Forms?

If you are studying a physical system, like a vibrating string or a chemical reaction, the equations are often coupled (everything depends on everything else). A canonical form decouples the system.

The most famous canonical form is the Diagonal Form. If a matrix $A$ is diagonalizable, it means there exists a basis where the operator simply scales each axis independently.

The Jordan Normal Form

What happens if a matrix is not diagonalizable? This occurs when there are not enough eigenvectors (the matrix is “defective”).

The Jordan Normal Form (JNF) is the best we can do for any square matrix. It decomposes the operator into Jordan Blocks on the diagonal: $J = \begin{pmatrix} \lambda & 1 & 0 \\ 0 & \lambda & 1 \\ 0 & 0 & \lambda \end{pmatrix}$

Everything off the block is zero. Inside the block, we have the eigenvalue on the diagonal and 1s just above it. These 1s represent “coupling” that cannot be removed.

Practical Use: Stability Analysis

In control theory, we look at the JNF to determine if a system will explode or settle.

• If the eigenvalues have negative real parts, the system is stable.
• If we have a Jordan block with $\lambda=0$ and a 1 above it, the system might grow linearly over time ($t$), which could be problematic!

import sympy as sp

# Define a 'defective' matrix (not diagonalizable)
A = sp.Matrix([[5, 4, 3], 
             [-1, 0, -3], 
             [1, -2, 1]])

# Compute Jordan Normal Form
P, J = A.jordan_form()

print("Original Matrix A:")
sp.pprint(A)

print("\nJordan Normal Form J:")
sp.pprint(J)

print("\nSimilarity Matrix P (Basis change):")
sp.pprint(P)

# Verify A = PJP^-1
assert A == P * J * P.inv()
print("\nVerification Successful: A = PJP^-1")

Rational Canonical Form

The Jordan form requires complex numbers (to find all roots of the characteristic polynomial). If we want to stay within the field of rational numbers $\mathbb{Q}$ or real numbers $\mathbb{R}$, we use the Rational Canonical Form (also known as the Frobenius map).

Instead of eigenvalues, this form uses Invariant Factors—polynomials that divide each other. This is deeply connected to the structure theory of modules over a Principal Ideal Domain (PID).

Exercises

Eigenvalues and Eigenvectors

Most vectors change direction when a linear transformation is applied. However, some special vectors keep their direction and are only stretched or shrunk. These are eigenvectors, and their scaling factor is the eigenvalue.

1. The Stability Equation

For a linear operator $A$, a vector $v$ is an eigenvector if: $Av = \lambda v$ where $\lambda$ is a scalar (the eigenvalue).

Intuition: In a 2D rotation, no real vector keeps its direction (except the zero vector). But in a scaling transformation, the axes are eigenvectors because points on them move only along the line.

import numpy as np
import matplotlib.pyplot as plt

A = np.array([[3, 1], 
            [0, 2]])

# v is an eigenvector [1, 0] with eigenvalue 3
v = np.array([1, 0])
Av = A @ v

# u is NOT an eigenvector [0, 1]
u = np.array([0, 1])
Au = A @ u

plt.quiver([0, 0], [0, 0], [v[0], u[0]], [v[1], u[1]], color=['b', 'k'], scale=5, label='Original')
plt.quiver([0, 0], [0, 0], [Av[0], Au[0]], [Av[1], Au[1]], color=['r', 'gray'], scale=5, label='Mapped')
plt.legend()
plt.title("v (blue) stays on line, u (black) changes direction")
plt.show()

2. Finding the Spectrum

To find $\lambda$, we must solve $\det(A - \lambda I) = 0$. This gives us the Characteristic Polynomial.

3. The Power Method: Finding Eigenvectors Iteratively

In high-dimensional spaces (like Google’s PageRank), we don’t calculate determinants. Instead, we use the Power Method: repeatedly apply $A$ to a random vector until it converges to the dominant eigenvector.

import numpy as np

A = np.array([[2, 1], 
            [1, 2]])
x = np.random.rand(2) # Start with random vector

for i in range(10):
  x = A @ x
  x = x / np.linalg.norm(x) # Normalize to prevent overflow
  print(f"Iteration {i}: {x}")

print(f"
Final estimate for dominant eigenvector: {x}")
print(f"Verify: A @ x = {A @ x}")

4. Diagonalization: $A = PDP^{-1}$

If a matrix has enough eigenvectors, we can rotate our coordinate system so the transformation is just axis-aligned scaling. This is the foundation of Principal Component Analysis (PCA).

5. Summary Check

Inner Product Spaces

Up to this point, our vector spaces have been “bare.” We can add vectors and scale them, but we have no notion of how long a vector is, or what the angle between two vectors might be. In bare linear algebra, there is no “perpendicular.”

Inner product spaces equip vector spaces with this geometric structure. This allows us to define limits, identify the “closest” approximation to a signal, and decompose data into independent components.

Geometry from Algebra

An inner product is a function that takes two vectors and returns a scalar. For $\mathbb{R}^n$, the standard inner product is the dot product $u \cdot v = \sum u_i v_i$. To generalize this to any vector space $V$ over $\mathbb{R}$ or $\mathbb{C}$, we define an inner product $\langle u, v \rangle$ as a map satisfying:

1. Conjugate Symmetry: $\langle u, v \rangle = \overline{\langle v, u \rangle}$.
2. Linearity: $\langle \alpha u + v, w \rangle = \alpha \langle u, w \rangle + \langle v, w \rangle$.
3. Positive Definiteness: $\langle v, v \rangle \geq 0$, and $\langle v, v \rangle = 0 \iff v = \mathbf{0}$.

Defining Length and Distance

The norm (length) of a vector is defined as $\|v\| = \sqrt{\langle v, v \rangle}$. This is the generalized Pythagorean theorem. The distance between two points is then simply $d(u, v) = \|u - v\|$.

The Cauchy-Schwarz Inequality

One of the most powerful results in mathematics is the Cauchy-Schwarz inequality: $|\langle u, v \rangle| \leq \|u\| \|v\|$ This ensures that the “angle” $\theta$ defined by $\cos \theta = \frac{\langle u, v \rangle}{\|u\| \|v\|}$ is always between -1 and 1 for real spaces.

Orthogonality: The Power of Perpendiculars

Two vectors are orthogonal if $\langle u, v \rangle = 0$. In terms of data, orthogonal vectors are completely “unrelated” or “uncorrelated” in the geometry defined by that inner product.

A set of vectors is orthonormal if they are all orthogonal to each other and all have length 1. Working with an orthonormal basis $\{e_1, \dots, e_n\}$ is trivial compared to a general basis because the coefficients of any vector $v$ are just the inner products: $v = \langle v, e_1 \rangle e_1 + \dots + \langle v, e_n \rangle e_n$

Gram-Schmidt: Generating Order from Chaos

The Gram-Schmidt process is a recipe for turning any basis into an orthonormal one. It works by taking the first vector, then taking the second and subtracting the part that “leaks” into the first, and so on.

import numpy as np

def gram_schmidt(B):
  """Orthonormalize the columns of matrix B."""
  Q = np.zeros_like(B, dtype=float)
  for i in range(B.shape[1]):
      v = B[:, i]
      # Subtract projections onto previously computed vectors
      for j in range(i):
          v = v - np.dot(Q[:, j], B[:, i]) * Q[:, j]
      # Normalize
      Q[:, i] = v / np.linalg.norm(v)
  return Q

# Example basis (columns of A)
A = np.array([[1.0, 1.0, 0.0], 
            [1.0, 0.0, 1.0], 
            [0.0, 1.0, 1.0]]).T

Q = gram_schmidt(A)
print("Original Basis:
", A)
print("
Orthonormal Basis (Q):
", Q)
print("
Check Orthogonality (Q.T @ Q):
", np.round(Q.T @ Q, 10))

Best Approximations and Projections

The most significant application of inner products is the Orthogonal Projection. Given a subspace $W$ and a vector $v$ not in $W$, the “closest” vector in $W$ to $v$ is the projection $P_W(v)$.

This is the engine behind Least Squares Regression. If we have a system $Ax = b$ that has no solution, we look for the $x$ that minimizes the error $\|Ax - b\|$. This happens when $Ax$ is the projection of $b$ onto the column space of $A$.

Fun Application: Fourier Series

Calculus students often see Fourier series as a specialized topic. In reality, it is just linear algebra on an infinite-dimensional inner product space of functions $L^2$. The functions $\sin(nx)$ and $\cos(nx)$ form an orthogonal basis. Calculating Fourier coefficients is exactly the same as calculating coordinates in $\mathbb{R}^n$ using dot products!

Exercises

Linear Independence and Span

Once we have a vector space, we need a way to describe its contents efficiently. If a vector space is a “playground,” then Linear Independence and Span are the rules for how many “tools” (vectors) you actually need to build everything in that playground.

1. The Concept of Span

The Span of a set of vectors $S = \{v_1, v_2, \dots, v_n\}$ is the set of all possible linear combinations of those vectors.

$\text{span}(S) = \{ c_1 v_1 + c_2 v_2 + \dots + c_n v_n : c_i \in \mathbb{R} \}$

Intuition: If you have two non-parallel arrows in a 2D plane, their span is the entire plane because you can reach any point by scaling and adding those two arrows.

import numpy as np
import matplotlib.pyplot as plt

v1 = np.array([1, 2])
v2 = np.array([2, 1])

# Generate random linear combinations (c1*v1 + c2*v2)
points = []
for _ in range(500):
  c1, c2 = np.random.uniform(-1, 1, 2)
  points.append(c1*v1 + c2*v2)

points = np.array(points)
plt.scatter(points[:,0], points[:,1], alpha=0.5, s=10)
plt.quiver([0,0], [0,0], [v1[0], v2[0]], [v1[1], v2[1]], color=['r','b'], scale=5)
plt.title("The Span of two non-parallel vectors (fills a 2D area)")
plt.grid(True)
plt.show()

2. Linear Independence: Avoiding Redundancy

A set of vectors is linearly independent if none of the vectors can be written as a linear combination of the others. In other words, they are all “original” and provide “new directions.”

Formal Definition

Vectors $\{v_1, \dots, v_n\}$ are linearly independent if the equation $c_1 v_1 + \dots + c_n v_n = \mathbf{0}$ has only the trivial solution $c_i = 0$.

If a vector can be built from others, the set is Dependent. We can test this by checking the Rank of the matrix formed by these vectors. If $\text{Rank}(A) < \text{Number of Vectors}$, they are dependent.

import numpy as np

# Case 1: Independent vectors
v1 = np.array([1, 0, 0])
v2 = np.array([0, 1, 0])
v3 = np.array([0, 0, 1])
mat1 = np.array([v1, v2, v3])
rank1 = np.linalg.matrix_rank(mat1)
print(f"Set 1 Rank: {rank1} (Independent if rank=3)")

# Case 2: Dependent vectors
u1 = np.array([1, 2, 3])
u2 = np.array([4, 5, 6])
u3 = u1 + u2 # Explicitly dependent
mat2 = np.array([u1, u2, u3])
rank2 = np.linalg.matrix_rank(mat2)
print(f"Set 2 Rank: {rank2} (Dependent if rank < 3)")

3. Visualizing Dependency in 3D

Imagine three vectors in 3D space. If one is a combination of the others, they all lie on a single plane (2D), even though there are three vectors.

4. Why Does It Matter?

Linear independence tells us if our data is redundant. If you have 100 sensors measuring the same physical phenomenon (e.g., temperature) and they are perfectly correlated, your “feature matrix” will be rank-deficient. You have 100 numbers, but effectively only 1 “dimension” of information.

import numpy as np

# Simulate 3 redundant sensors
temp = np.random.normal(25, 1, 100) # True temperature
s1 = temp + 0.1
s2 = 2 * temp - 5
s3 = temp * 0.5 + 10

data = np.vstack([s1, s2, s3]).T
rank = np.linalg.matrix_rank(data)

print(f"Data shape: {data.shape}")
print(f"Calculated Rank: {rank}")
print("Even though we have 3 sensors, the rank is 1 because they are all linear scalings of each other.")

5. Summary Check

Matrices and Linear Systems

Matrices are the numerical engines of Linear Algebra. While a “Vector Space” is a theoretical playground, a Matrix is the specific blueprint that tells us how to manipulate that space.

1. The Matrix as a Map

A matrix $A$ of size $m \times n$ is a grid of numbers that represents a mapping from $\mathbb{R}^n$ to $\mathbb{R}^m$. Each column of the matrix tells us where one of the basis vectors of $\mathbb{R}^n$ “lands” in $\mathbb{R}^m$.

Let’s visualize a Shear Transformation matrix: $A = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$. This matrix leaves the $x$-axis alone but shifts the $y$-direction.

import numpy as np
import matplotlib.pyplot as plt

# A 2x2 Shear Matrix
A = np.array([[1, 1], 
            [0, 1]])

# Create a grid of points (a square)
x, y = np.meshgrid(np.linspace(0, 1, 10), np.linspace(0, 1, 10))
points = np.vstack([x.flatten(), y.flatten()])

# Transform the points: A * points
transformed = A @ points

plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.scatter(points[0], points[1], c='blue', s=5)
plt.title("Original Square")
plt.axis('equal')

plt.subplot(1, 2, 2)
plt.scatter(transformed[0], transformed[1], c='red', s=5)
plt.title("Transformed (Shear)")
plt.axis('equal')
plt.show()

2. Linear Systems: $Ax = b$

A system of linear equations asks: “Which vector $x$ lands on $b$ when we apply the transformation $A$?”

If the matrix $A$ squashes space into a lower dimension (i.e., it is Rank-Deficient or Singular), then $b$ might be unreachable, or there might be infinitely many paths to reach it.

import numpy as np

# A singular matrix (rows are multiples)
A = np.array([[1, 2], 
            [2, 4]])
b = np.array([5, 10])

try:
  x = np.linalg.solve(A, b)
  print(f"Solution: {x}")
except np.linalg.LinAlgError:
  print("Matrix is singular! No unique solution.")

# Check the rank
print(f"Rank of A: {np.linalg.matrix_rank(A)}")
print("Since Rank (1) < Dimensions (2), the matrix isn't 'full' and can't be inverted.")

3. Multiplication: Composition of Maps

Multiplying two matrices $AB$ represents applying transformation $B$ first, then $A$. Because the order of geometric transformations (like rotating then shifting) matters, matrix multiplication is not commutative ($AB \neq BA$).

4. Reduced Row Echelon Form (RREF)

RREF is the “simplest” version of a matrix that still represents the same linear system. It allows us to read off the solutions directly. In RREF, the leading entry of each row is 1, and all other entries in that column are 0.

5. Summary Check

Representation Theory

Group theory studies symmetry in the abstract. Linear algebra studies matrices acting on vectors. Representation Theory is the bridge between them: it studies how abstract groups can be represented as matrices. This allows us to use the powerful tools of linear algebra (trace, determinant, eigenvalues) to solve problems in abstract algebra and physics.

The Basic Idea

A collection of symmetries (a Group $G$) can often be represented by a set of linear transformations on a vector space $V$. A representation is a homomorphism $\rho: G \to GL(V)$. This means that for every group element $g$, there is an invertible matrix $\rho(g)$ such that the group composition is preserved: $\rho(g_1 g_2) = \rho(g_1) \rho(g_2)$

Irreducible Representations (Irreps)

Just as an integer can be broken down into prime factors, a representation can often be broken down into smaller, simpler representations. If a representation cannot be broken down further, it is called irreducible.

The Maschke Theorem states that for finite groups (over fields like $\mathbb{C}$), every representation is a direct sum of irreducible ones. This is effectively the “fundamental theorem of arithmetic” for representations.

Characters: Data Compression for Symmetries

Working with full matrices for every group element is computationally expensive. Character Theory simplifies this by focusing only on the trace of the matrices. The character of a representation $\rho$ is the function $\chi: G \to \mathbb{C}$ defined by: $\chi(g) = \text{tr}(\rho(g))$

Characters are “class functions”—they are the same for elements in the same conjugacy class. This remarkably compact representation contains almost all the information about the representation.

import numpy as np

# Representation of the Cyclic Group C4 
# Generated by a rotation of 90 degrees
def get_c4_rep(k):
  # k is the power (0, 1, 2, 3)
  theta = k * (np.pi / 2)
  return np.array([[np.cos(theta), -np.sin(theta)],
                   [np.sin(theta),  np.cos(theta)]])

# Calculate characters for each element
characters = []
for k in range(4):
  rho_g = get_c4_rep(k)
  chi_g = np.trace(rho_g)
  characters.append(np.round(chi_g, 2))

print("Group Elements: [e, r, r^2, r^3]")
print("Characters:    ", characters)

# Note: The trace of the identity (e) is 2, 
# which is the dimension of the representation.

Symmetry in Physics and Chemistry

In quantum mechanics, particles are described by wavefunctions. If a physical system has a certain symmetry (like rotating a crystal), the wavefunction must transform according to a representation of that symmetry group.

• Molecular Vibrations: Predicting which vibrations are “infrared active” in a molecule is done by decomposing the representation of the molecule’s symmetry group.
• Particle Physics: The “Standard Model” is built on the representations of the groups $SU(3) \times SU(2) \times U(1)$. Particles are literally just vectors in the spaces where these groups act!

Exercises

Subspaces and Quotients

A vector space can contain smaller vector spaces called subspaces. We can also “divide” a space by a subspace to create a quotient space.

Subspaces

A subset $W$ of a vector space $V$ is a subspace if:

1. The zero vector $\mathbf{0} \in W$.
2. $W$ is closed under addition: $\mathbf{u}, \mathbf{v} \in W \implies \mathbf{u} + \mathbf{v} \in W$.
3. $W$ is closed under scalar multiplication: $c \in F, \mathbf{v} \in W \implies c\mathbf{v} \in W$.

import numpy as np

def is_in_subspace(vector, basis):
  """Check if a vector is in the subspace spanned by a basis."""
  # Solve basis * c = vector
  A = np.array(basis).T
  try:
      x, residuals, rank, s = np.linalg.lstsq(A, vector, rcond=None)
      # If residuals are near zero, it's in the subspace
      return np.allclose(np.dot(A, x), vector)
  except:
      return False

subspace_basis = [[1, 0, 0], [0, 1, 0]]
v1 = [0.5, 2.0, 0.0]
v2 = [0.5, 2.0, 1.0]

print(f"Vector {v1} in subspace: {is_in_subspace(v1, subspace_basis)}")
print(f"Vector {v2} in subspace: {is_in_subspace(v2, subspace_basis)}")

Quotient Spaces

Given a subspace $W \subseteq V$, the quotient space $V/W$ is the set of all cosets $\mathbf{v} + W$. Intuitively, the quotient space “ignores” all differences that lie within $W$.

The dimension of a quotient space is: $\dim(V/W) = \dim(V) - \dim(W)$

Kernels and Images

For a linear map $T: V \to W$:

• The Kernel $\ker(T) = \{ \mathbf{v} \in V \mid T(\mathbf{v}) = \mathbf{0} \}$ is a subspace of $V$.
• The Image $im(T) = \{ T(\mathbf{v}) \mid \mathbf{v} \in V \}$ is a subspace of $W$.

import numpy as np
from scipy.linalg import null_space

A = np.array([[1, 2, 3], [2, 4, 6], [1, 1, 1]])

# Find basis for kernel
ns = null_space(A)
print(f"Basis for Kernel (Null Space):\n{ns}")

# Find rank (dimension of image)
rank = np.linalg.matrix_rank(A)
print(f"Rank (dim of Image): {rank}")

Singular Value Decomposition (SVD)

The Singular Value Decomposition (SVD) is arguably the most important result in applied linear algebra. While eigendecomposition works only for special square matrices, SVD works for any matrix—tall, wide, square, singular, or non-singular. It provides a way to “see” the underlying structure of data by decomposing it into its most significant components.

The Geometric Idea

Every matrix $A$ represents a linear map. SVD says that any such map can be broken down into three simple steps:

1. A rotation in the input space ($V$).
2. A scaling along the principal axes ($\Sigma$).
3. A rotation in the output space ($U$).

Mathematically: $A = U \Sigma V^T$

• $V$: Columns are “right singular vectors.” They define an orthonormal basis in the input space.
• $\Sigma$: A diagonal matrix of “singular values” $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_n \ge 0$. These tell you the “strength” or “gain” of the matrix in each direction.
• $U$: Columns are “left singular vectors.” They define an orthonormal basis in the output space.

Data Compression: The Best Low-Rank Approximation

The real magic of SVD is the Eckart-Young Theorem. It states that if you want the best possible “summary” of a matrix $A$ using only $k$ dimensions (where $k$ is less than the rank of $A$), the answer is to keep only the $k$ largest singular values and their corresponding vectors.

$A_k = \sum_{i=1}^k \sigma_i u_i v_i^T$

This is how image compression and noise reduction work. By throwing away small singular values, we lose “noise” or “unimportant detail” but keep the overall structure.

import numpy as np

# Create a 'rank-2' matrix with some noise
# This matrix has 100 rows (data points) and 5 columns (features)
np.random.seed(42)
base_data = np.random.randn(100, 2) @ np.random.randn(2, 5)
noise = 0.1 * np.random.randn(100, 5)
A = base_data + noise

# Perform SVD
U, s, Vt = np.linalg.svd(A, full_matrices=False)

print("Singular Values:", s)

# Construct Rank-2 approximation
k = 2
Sk = np.diag(s[:k])
Ak = U[:, :k] @ Sk @ Vt[:k, :]

# Calculate Error
full_norm = np.linalg.norm(A, 'fro')
error_norm = np.linalg.norm(A - Ak, 'fro')

print(f"\nRelative Error of Rank-{k} approximation: {error_norm/full_norm:.4f}")
print(f"Original Rank: {np.linalg.matrix_rank(A)}")
print(f"Approximation Rank: {np.linalg.matrix_rank(Ak)}")

The Pseudoinverse: Solving the Solvable and Unsolvable

When a matrix $A$ is not invertible (e.g., it is not square or is singular), we can still “solve” $Ax = b$ using the Moore-Penrose Pseudoinverse $A^\dagger$.

Using SVD, the pseudoinverse is trivial to compute: $A^\dagger = V \Sigma^\dagger U^T$ where $\Sigma^\dagger$ is formed by transposing $\Sigma$ and replacing every non-zero $\sigma_i$ with $1/\sigma_i$.

The solution $x = A^\dagger b$ is the “best” solution in two senses:

1. It minimizes the error $\|Ax - b\|^2$ (Least Squares).
2. If there are many such solutions, it picks the one with the smallest length $\|x\|$.

Exercises

Tensors and Multilinear Algebra

A scalar is a “rank-0” tensor (a single number). A vector is a “rank-1” tensor (an array). A matrix is a “rank-2” tensor (a grid). Beyond these lie higher-rank tensors—multi-dimensional arrays that follow specific transformation rules. Tensors are the natural language of General Relativity, Quantum Mechanics, and modern Deep Learning.

What is a Tensor?

While computer scientists often define a tensor as “a multidimensional array,” mathematicians define it by how it transforms or what it does.

A tensor is a multilinear map. Just as a matrix $A$ represents a linear map $f(v)$, a tensor $T$ takes multiple vectors as input and produces a scalar (or another vector). For example, a rank-2 tensor $T(u, v)$ is a function that is linear in $u$ AND linear in $v$.

Covariance and Contravariance

In physics, we distinguish between:

• Contravariant vectors ($v^i$): Things like velocity or position that scale “with” the coordinate system.
• Covariant vectors ($w_i$): Things like gradients or dual vectors that scale “against” the coordinate changes.

A general tensor $T^i_j$ can have both types of indices. This distinction is crucial for ensuring that physical laws remain the same regardless of the units or axes we choose.

The Tensor Product

The tensor product $\otimes$ is a way to combine two vector spaces $V$ and $W$ into a larger space $V \otimes W$. If $v \in V$ and $w \in W$, then $v \otimes w$ is an element of the product space. If $V = \mathbb{R}^m$ and $W = \mathbb{R}^n$, the tensor product $V \otimes W$ has dimension $m \times n$. You can think of this as the space of all possible $m \times n$ matrices.

import numpy as np

# Create two vectors
v = np.array([1, 2, 3])
w = np.array([4, 5])

# Compute the Outer Product (a simple tensor product)
# This results in a Rank-2 tensor (Matrix)
T = np.outer(v, w)

print("Vector v (Rank 1):", v)
print("Vector w (Rank 1):", w)
print("\nTensor Product v ⊗ w (Rank 2 Matrix):\n", T)

# Higher-order tensor via expansion
# A Rank-3 tensor could be representing interactions 
# between three spaces.
T3 = np.zeros((3, 2, 2))
T3[0, :, :] = np.eye(2)
T3[1, :, :] = np.ones((2, 2))
T3[2, :, :] = np.zeros((2, 2))

print("\nRank-3 Tensor (3x2x2):\n", T3)
print("\nAccessing a slice T3[0]:\n", T3[0])

Multilinear Forms and Determinants

The determinant is the most famous example of a multilinear form. It is an alternating $(n,0)$-tensor. “Alternating” means that if you swap two input vectors, the sign of the output flips. This property is what allows the determinant to measure signed volume.

Einstein Summation Notation

In tensor calculus, we often drop the summation sign $\sum$. If an index appears twice (once up, once down), it is summed over. Instead of: $y^i = \sum_j A^i_j x^j$ We write: $y^i = A^i_j x^j$ This compact notation is the standard in engineering and theoretical physics.

Exercises

Linear Transformations and Matrices

A linear transformation $T: V \to W$ is a mapping between vector spaces that preserves the operations of addition and scalar multiplication.

Properties of Linear Maps

$T$ is linear if for all $\mathbf{u}, \mathbf{v} \in V$ and $c \in F$:

1. $T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})$
2. $T(c\mathbf{v}) = cT(\mathbf{v})$

Every linear transformation between finite-dimensional vector spaces can be represented as a matrix.

import numpy as np

def apply_transformation(A, v):
  return np.dot(A, v)

# Rotation Matrix by 90 degrees (pi/2)
theta = np.pi / 2
R = np.array([[np.cos(theta), -np.sin(theta)],
            [np.sin(theta),  np.cos(theta)]])

v = np.array([1, 0])
v_rot = apply_transformation(R, v)

print(f"Original: {v}")
print(f"Rotated 90 deg: {v_rot.round(2)}")

Composition and Matrix Multiplication

If $T: U \to V$ and $S: V \to W$ are linear maps, their composition $S \circ T: U \to W$ is also linear. The matrix representing $S \circ T$ is the product of the matrices representing $S$ and $T$.

Change of Basis

The matrix representation of a linear map depends on the choice of bases for $V$ and $W$. If $[T]_B$ is the matrix in basis $B$, and $P$ is the transition matrix from basis $B'$ to $B$, then: $[T]_{B'} = P^{-1} [T]_B P$

# T in standard basis
T_std = np.array([[1, 2], [3, 4]])

# New basis B' = {[1,1], [1,0]}
P = np.array([[1, 1], [1, 0]])
P_inv = np.linalg.inv(P)

# T in new basis
T_new = P_inv @ T_std @ P
print(f"Matrix in new basis:\n{T_new.round(2)}")

The Axiomatic Foundation of Vector Spaces

In elementary physics, a vector is often described as a directed line segment. While intuitive, this definition is insufficient for higher mathematics. Modern linear algebra treats a Vector Space as an abstract algebraic structure—a “playground” where elements can be added together and scaled by numbers.

1. Defining the Playground

A Vector Space $V$ over a field $F$ (typically $\mathbb{R}$) is a set equipped with two operations: vector addition ($+$) and scalar multiplication ($\cdot$). Instead of memorizing axioms as dry rules, we can view them as the “laws of physics” for our data.

Standard Euclidean Space: $\mathbb{R}^n$

The most common example is $\mathbb{R}^n$, where addition and scaling are performed component-wise. Let’s verify the Commutativity ($u + v = v + u$) and Distributivity ($a(u+v) = au + av$) properties using NumPy.

import numpy as np

# Define two vectors in R3
u = np.array([1, 2, 3])
v = np.array([4, 5, 6])
a = 2.5

# 1. Verify Commutativity
lhs_add = u + v
rhs_add = v + u
print(f"Addition Commutativity: {lhs_add} == {rhs_add}")

# 2. Verify Distributivity
lhs_dist = a * (u + v)
rhs_dist = a * u + a * v
print(f"Distributivity: {lhs_dist} == {rhs_dist}")

# Check if they are numerically equal
print(f"Equal? {np.allclose(lhs_dist, rhs_dist)}")

2. The Eight Axioms

For any set $V$ to be a formal Vector Space, these eight rules must hold for all $u, v, w \in V$ and scalars $a, b \in \mathbb{R}$:

1. Commutativity: $u + v = v + u$.
2. Associativity: $(u + v) + w = u + (v + w)$.
3. Additive Identity: There is a $\mathbf{0}$ such that $v + \mathbf{0} = v$.
4. Additive Inverse: For every $v$, there is a $-v$ such that $v + (-v) = \mathbf{0}$.
5. Multiplicative Identity: $1 \cdot v = v$.
6. Compatibility: $a(bv) = (ab)v$.
7. Distributivity of Scalar: $a(u + v) = au + av$.
8. Distributivity of Vector: $(a + b)v = av + bv$.

Exercise: The Non-Vector Space

Consider the set of all points in the first quadrant: $Q = \{(x, y) \in \mathbb{R}^2 \mid x, y \ge 0\}$. Why does this fail to be a vector space?

import numpy as np

# A point in the first quadrant
v = np.array([5, 10])

# Scaling by a negative number
scalar = -1
result = scalar * v

print(f"Original vector: {v}")
print(f"Scaled by -1: {result}")

# Is result in the first quadrant?
is_in_q = np.all(result >= 0)
print(f"Is the result still in the first quadrant? {is_in_q}")

The failure shown above violates Closure under Scalar Multiplication. If we scale a “positive” vector by a negative number, we leave the set. Thus, the first quadrant is not a vector space.

3. Abstract Examples: Polynomials

The beauty of these axioms is that “vectors” don’t have to be arrows. They can be functions or polynomials. The set $\mathbb{P}_n$ of polynomials of degree $\le n$ forms a vector space because adding two polynomials yields another polynomial, and the axioms hold.

4. Function Spaces

In advanced applications like Fourier analysis, we treat signals (functions) as vectors. If $f(t)$ and $g(t)$ are continuous functions, then $h(t) = f(t) + g(t)$ is also continuous.

import numpy as np
import matplotlib.pyplot as plt

t = np.linspace(0, 1, 100)
f = np.sin(2 * np.pi * t)
g = np.cos(4 * np.pi * t)

# Vector addition of 'functions' (sampled)
h = f + g

plt.plot(t, f, '--', label='f(t)')
plt.plot(t, g, '--', label='g(t)')
plt.plot(t, h, 'k', linewidth=2, label='f(t) + g(t)')
plt.legend()
plt.title("Addition in Function Space")
plt.show()

5. Summary Check