Eigenvalues and singular values | Nicholas Hu

The Schur decomposition

Wed, 28 May 2025 00:00:00 +0000

$$ %% Sets and functions %% \newcommand{\set}[1]{\{ #1 \}} \newcommand{\Set}[1]{\left \{ #1 \right\}} \renewcommand{\emptyset}{\varnothing} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} \newcommand{\R}{\mathbb{R}} \newcommand{\Rn}{\mathbb{R}^n} \newcommand{\Rm}{\mathbb{R}^m} \newcommand{\C}{\mathbb{C}} \newcommand{\F}{\mathbb{F}} %% Linear algebra %% \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\Abs}[1]{\left\lvert #1 \right\rvert} \newcommand{\inner}[2]{\langle #1, #2 \rangle} \newcommand{\Inner}[2]{\left\langle #1, #2 \right\rangle} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\Norm}[1]{\left\lVert #1 \right\rVert} \newcommand{\tp}{{\top}} \newcommand{\trans}{{\top}} \newcommand{\span}{\operatorname{span}} \newcommand{\im}{\operatorname{im}} \newcommand{\ker}{\operatorname{ker}} \newcommand{\rank}{\operatorname{rank}} \newcommand{\proj}{\operatorname{proj}} \newcommand{\proj}[1]{\mathop{\mathrm{proj}_{#1}}} \newcommand{\refl}{\operatorname{refl}} \newcommand{\refl}[1]{\mathop{\mathrm{refl}_{#1}}} \newcommand{\K}{\mathcal{K}} \newcommand{\L}{\mathcal{L}} \renewcommand{\epsilon}{\varepsilon} \newcommand{\conj}{\overline} \newcommand{\sign}{\operatorname{sign}} %% Colours %% \definecolor{cblue}{RGB}{31, 119, 180} \definecolor{corange}{RGB}{255, 127, 14} \definecolor{cgreen}{RGB}{44, 160, 44} \definecolor{cred}{RGB}{214, 39, 40} \definecolor{cpurple}{RGB}{148, 103, 189} \definecolor{cbrown}{RGB}{140, 86, 75} \definecolor{cpink}{RGB}{227, 119, 194} \definecolor{cgrey}{RGB}{127, 127, 127} \definecolor{cyellow}{RGB}{188, 189, 34} \definecolor{cteal}{RGB}{23, 190, 207} $$

The complex Schur decomposition

Let $A \in \C^{n \times n}$. The (complex) Schur decomposition is a factorization of $A$ as $UTU^{-1}$, where $U \in \C^{n \times n}$ is unitary and $T \in \C^{n \times n}$ is upper triangular.

Such a factorization always exists and can be constructed recursively: let $\lambda \in \C$ be an eigenvalue of $A$ (which exists by the fundamental theorem of algebra) and $v_1 \in \C^n$ be a corresponding normalized eigenvector. Extending $\set{v_1}$ to an orthonormal basis $\set{v_j}_{j=1}^n$ and defining $V := \begin{bmatrix} v_1 & \cdots & v_n \end{bmatrix} \in \C^{n \times n}$, we obtain $$ A = V \begin{bmatrix} \lambda & b^* \\ & \hat{A} \end{bmatrix} V^{-1} $$ for some $\hat{A} \in \C^{(n-1) \times (n-1)}$ and $b \in \C^{n-1}$. Thus, if $\hat{A}$ has a Schur decomposition $\hat{U} \hat{T} \hat{U}^{-1}$, then $$ A = \underbrace{V \begin{bmatrix} 1 & \\ & \hat{U} \end{bmatrix}}_{U} \underbrace{\begin{bmatrix} \lambda & b^* \hat{U} \\ & \hat{T} \end{bmatrix}}_T \underbrace{\begin{bmatrix} 1 & \\ & \hat{U}^{-1} \end{bmatrix} V^{-1}}_{U^{-1}} $$ is a Schur decomposition of $A$, and its existence in the base case $n = 1$ is trivial.

Complex normal matrices

If a normal matrix $T \in \C^{n \times n}$ can be partitioned as $\begin{bmatrix} T_{11} & T_{12} \\ & T_{22} \end{bmatrix}$ with $T_{11} \in \C^{m \times m}$, then by definition $$ \begin{bmatrix} T_{11} & T_{12} \\ & T_{22} \end{bmatrix} \begin{bmatrix} T_{11}^* & \\ T_{12}^* & T_{22}^* \end{bmatrix} = \begin{bmatrix} T_{11}^* & \\ T_{12}^* & T_{22}^* \end{bmatrix} \begin{bmatrix} T_{11} & T_{12} \\ & T_{22} \end{bmatrix}, $$ so $T_{11} T_{11}^* + T_{12} T_{12}^* = T_{11}^* T_{11}$. Taking traces, we obtain $\norm{T_{11}}_\mathrm{F}^2 + \norm{T_{12}}_\mathrm{F}^2 = \norm{T_{11}}_\mathrm{F}^2$, which implies that $T_{12} = 0$ and hence that $T_{11}$ and $T_{22}$ are normal.

Now if $A$ is normal, then so is the upper triangular matrix $T = U^* A U$ in its complex Schur factorization. It follows that $T$ must in fact be diagonal, which yields the complex spectral theorem: a (square) complex matrix is unitarily diagonalizable if and only if it is normal.

The real Schur decomposition

Let $A \in \R^{n \times n}$. The real Schur decomposition is a factorization of $A$ as $QTQ^{-1}$, where $Q \in \R^{n \times n}$ is orthogonal and $T \in \R^{n \times n}$ is upper quasi-triangular – that is, block upper triangular with diagonal blocks that are $1 \times 1$ or $2 \times 2$.

As in the complex case, such a factorization always exists; arguing as above, it suffices to show that $A$ has a one- or two-dimensional invariant subspace. Indeed, when regarded as a complex matrix, $A$ has an eigenvalue $\lambda = \alpha + i \beta \in \C$ (where $\alpha, \beta \in \R$) and an eigenvector $v_1 = x_1 + iy_1 \in \C^n$ (where $x_1, y_1 \in \R^n$).

If $\lambda \in \R$, then $Ax_1 + i Ay_1 = \lambda x_1 + i \lambda y_1$, so $x_1$ or $y_1$ is an eigenvector with eigenvalue $\lambda$ (at least one of these vectors must be nonzero) and therefore spans a one-dimensional invariant subspace of $A$. On the other hand, if $\lambda \in \C \setminus \R$, then $Ax_1 + i Ay_1 = (\alpha x_1 - \beta y_1) + i(\beta x_1 + \alpha y_1)$. Moreover, $\conj{v_1}$ is an eigenvector of $A$ with eigenvalue $\conj{\lambda}$ and $\lambda \neq \conj{\lambda}$, so $\set{v_1, \conj{v_1}}$ is linearly independent. Hence $x_1 = \frac{1}{2}(v_1 + \conj{v_1})$ and $y_1 = \frac{1}{2i}(v_1 - \conj{v_1})$ span a two-dimensional invariant subspace of $A$.

Real normal matrices

If $A$ is normal, then $T = Q^* A Q$ will be “quasi-diagonal” – block diagonal with diagonal blocks that are $1 \times 1$ or $2 \times 2$ – and each diagonal block will itself be normal. Thus, these blocks will be of the form $$ \begin{bmatrix} \alpha \end{bmatrix} \quad (\alpha \in \R) \qquad \text{or} \qquad \begin{bmatrix} \alpha & \beta \\ -\beta & \alpha \end{bmatrix} \quad (\alpha \in \R,\, \beta \in \R \setminus \set{0}). $$ In particular, if $A$ is symmetric, there will only be $1 \times 1$ blocks, which yields the real spectral theorem: a (square) real matrix is orthogonally diagonalizable if and only if it is symmetric. If $A$ is orthogonal, we will have $\alpha = \pm 1$ in the $1 \times 1$ blocks and $\alpha = \cos(\theta)$ and $\beta = \sin(\theta)$ for some $\theta \in \R \setminus \pi \Z$ in the $2 \times 2$ blocks.

The singular value decomposition

Thu, 09 Jan 2025 00:00:00 +0000

PDF

$$ %% Sets and functions %% \newcommand{\set}[1]{\{ #1 \}} \newcommand{\Set}[1]{\left \{ #1 \right\}} \renewcommand{\emptyset}{\varnothing} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} \newcommand{\R}{\mathbb{R}} \newcommand{\Rn}{\mathbb{R}^n} \newcommand{\Rm}{\mathbb{R}^m} \newcommand{\C}{\mathbb{C}} \newcommand{\F}{\mathbb{F}} %% Linear algebra %% \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\Abs}[1]{\left\lvert #1 \right\rvert} \newcommand{\inner}[2]{\langle #1, #2 \rangle} \newcommand{\Inner}[2]{\left\langle #1, #2 \right\rangle} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\Norm}[1]{\left\lVert #1 \right\rVert} \newcommand{\trans}{{\top}} \newcommand{\span}{\mathop{\mathrm{span}}} \newcommand{\im}{\mathop{\mathrm{im}}} \newcommand{\ker}{\mathop{\mathrm{ker}}} \newcommand{\rank}{\mathop{\mathrm{rank}}} %% Colours %% \definecolor{cblue}{RGB}{31, 119, 180} \definecolor{corange}{RGB}{255, 127, 14} \definecolor{cgreen}{RGB}{44, 160, 44} \definecolor{cred}{RGB}{214, 39, 40} \definecolor{cpurple}{RGB}{148, 103, 189} \definecolor{cbrown}{RGB}{140, 86, 75} \definecolor{cpink}{RGB}{227, 119, 194} \definecolor{cgrey}{RGB}{127, 127, 127} \definecolor{cyellow}{RGB}{188, 189, 34} \definecolor{cteal}{RGB}{23, 190, 207} $$

Let $A \in \C^{m \times n}$. The singular value decomposition (SVD) is a factorization of $A$ as $U \Sigma V^*$, where $U \in \C^{m \times m}$ and $V \in \C^{n \times n}$ are unitary and $\Sigma \in \R^{m \times n}$ is (rectangular) diagonal with nonnegative entries. ¹ In other words, $A = \sum_{i=1}^{\min \set{m, n}} \sigma_i u_i v_i^*$, where $u_i$ and $v_i$ are the $i$^th columns of $U$ and $V$ and $\sigma_i$ is the $i$^th diagonal entry of $\Sigma$. The vectors $u_i$ and $v_i$ are called left and right singular vectors of $A$ and the scalars $\sigma_i$ are called singular values of $A$; by convention, we arrange the singular values in decreasing order.

If an SVD of $A$ has $r$ nonzero singular values, then $\set{u_i}_{i=1}^r$ is an orthonormal basis of $\im(A)$ because $Av_i = \sigma_i u_i$ for all $i$. Hence $r$ must be the rank of $A$ and $\set{u_i}_{i=r+1}^m$ an orthonormal basis of $\ker(A^*)$; similarly, $\set{v_i}_{i=1}^r$ and $\set{v_i}_{i=r+1}^n$ are orthonormal bases of $\im(A^*)$ and $\ker(A)$.

Existence

Assume without loss of generality that $m \geq n$. Clearly, the matrix $A^* A$ is (Hermitian) positive semidefinite, so by the spectral theorem, $A^* A = V \Lambda V^*$ for some unitary $V \in \C^{n \times n}$ and some diagonal $\Lambda \in \R^{n \times n}$ with diagonal entries $\lambda_1 \geq \cdots \geq \lambda_n \geq 0$. Set $\sigma_i = \sqrt{\lambda_i}$ for each $i$ and $Av_i = \sigma_i u_i$ for each nonzero $\sigma_i$. If $r$ is as above, $\hat{U} := \begin{bmatrix} u_1 & \cdots & u_r \end{bmatrix} \in \C^{m \times r}$, and $\hat{\Sigma} := \mathrm{diag}(\sigma_1, \dots, \sigma_r) \in \R^{r \times r}$, then by construction $$ AV = \hat{U} \begin{bmatrix} \hat{\Sigma} & 0_{r \times (n-r)} \end{bmatrix}. $$ Moreover, $\inner{u_i}{u_j} = \inner{Av_i / \sigma_i}{Av_j / \sigma_j} = \inner{\lambda_i v_i}{v_j} / \sigma_i \sigma_j = \delta_{ij}$, so $\set{u_i}_{i=1}^r$ is orthonormal. Extending this set to an orthonormal basis $\set{u_i}_{i=1}^m$, and defining $U = \begin{bmatrix} u_1 & \cdots & u_m \end{bmatrix} \in \C^{m \times m}$ and $$ \Sigma = \begin{bmatrix} \hat{\Sigma} & 0_{r \times (n-r)} \\ 0_{(m-r) \times r} & 0_{(m-r) \times (n-r)}\end{bmatrix} \in \R^{m \times n}, $$ we obtain $A = U \Sigma V^*$ as required. ²

Although an SVD is not unique, this argument shows that the singular values are unique and that the singular vectors are unique up to complex signs if $m = n$ and the singular values are distinct, since we must have $A^* A = V (\Sigma^* \Sigma) V^*$.

Low-rank approximation

Eckart–Young theorem

Suppose that $U \Sigma V^*$ is an SVD of a matrix $A \in \C^{m \times n}$ with rank $r$. If $k \leq r$ and $A_k := \sum_{i=1}^k \sigma_i u_i v_i^*$, then $\norm{A - B}_2 \geq \sigma_{k+1} = \norm{A - A_k}_2$ for all $B \in \C^{m \times n}$ such that $\rank(B) \leq k$ (where $\sigma_{r+1} := 0$). In particular, $\norm{A}_2 = \sigma_1$.

Proof. Suppose that $B \in \C^{m \times n}$ is such that $\rank(B) \leq k$. Then $\dim(\ker(B)) \geq n-k$, so there exists a $v \in \ker(B) \cap \span \set{v_i}_{i=1}^{k+1}$ such that $\norm{v}_2 = 1$. Hence $\norm{A - B}_2^2 \geq \norm{(A-B)v}_2^2 = \norm{Av}_2^2 = \sum_{i=1}^{k+1} \sigma_i^2 \abs{\inner{v}{v_i}}^2 \geq \sigma_{k+1}^2$. Similarly, if $v \in \C^n$ with $\norm{v}_2 = 1$, then $\norm{(A-A_k) v}_2^2 = \sum_{i=k+1}^r \sigma_i^2 \abs{\inner{v}{v_i}}^2 \leq \sigma_{k+1}^2$, with equality if $v = v_{k+1}$. ∎

An analogous theorem holds for the Frobenius norm (which can be proven similarly). In fact, we have the following generalization.

Eckart–Young–Mirsky theorem

Suppose that $U \Sigma V^*$ is an SVD of a matrix $A \in \C^{m \times n}$ with rank $r$ and let $\norm{{}\cdot{}}$ be a unitarily invariant norm. If $k \leq r$ and $A_k := \sum_{i=1}^k \sigma_i u_i v_i^*$, then $\norm{A - B} \geq \norm{A - A_k}$ for all $B \in \C^{m \times n}$ such that $\rank(B) \leq k$.

Proof. We begin by proving Weyl’s inequality for singular values: $$ \sigma_{i+j-1}(A+B) \leq \sigma_i(A) + \sigma_j(B), $$ where $\sigma_i({}\cdot{})$ denotes the $i$^th singular value of a given matrix. Let $A_k := \sum_{i=1}^k \sigma_i(A) u_i v_i^*$. Then $\rank(A_{i-1} + B_{j-1}) \leq (i-1) + (j-1) = i+j-2$, so by the Eckart–Young theorem, $\sigma_{i+j-1}(A+B) \leq \norm{(A + B) - (A_{i-1} + B_{j-1})}_2 \leq \norm{A - A_{i-1}}_2 + \norm{B - B_{j-1}}_2 = \sigma_i(A) + \sigma_j(B)$.

Now suppose that $B \in \C^{m \times n}$ is such that $\rank(B) \leq k$. By Weyl’s inequality, $\sigma_{k+i}(A) \leq \sigma_{k+1}(B) + \sigma_i(A-B) = \sigma_i(A-B)$ for all $i$ (where $\sigma_{k+i}({}\cdot{}) := 0$ if $k + i > \min \set{m, n}$). Thus, by unitary invariance, it suffices to show that $\Phi(x) := \norm{\operatorname{diag}(x_1, x_2, \dots)} \leq \Phi(y)$ whenever $0 \leq x \leq y$ componentwise; by induction and permutation invariance, we may assume without loss of generality that $x_i = y_i$ for all $i > 1$. ³ Accordingly, let $\theta \in [0, 1]$ be such that $x_1 = \theta y_1$. Then $\Phi(x) = \Phi(\frac{1+\theta}{2} y_1 + \frac{1-\theta}{2}(-y_1), y_2, \dots) \leq \frac{1+\theta}{2} \Phi(y_1, y_2, \dots) + \frac{1-\theta}{2} \Phi(-y_1, y_2, \dots) = \Phi(y)$, as was to be shown. ∎

If $A \in \R^{m \times n}$, an SVD is defined analogously; i.e., with $U$ and $V$ orthogonal. ↩︎
If we instead add $n-r$ rows of zeroes to $\begin{bmatrix} \hat{\Sigma} & 0 \end{bmatrix}$, forming a square matrix, the resulting decomposition is sometimes called the thin SVD; if we instead omit the last $n-r$ columns, what remains is sometimes called the compact SVD. ↩︎
Clearly, if $A \mapsto \Phi(\sigma_1(A), \sigma_2(A), \dots)$ is a unitarily invariant norm, then $\Phi$ is a symmetric gauge function: a norm on a real coordinate space that is permutation invariant and absolute ( $\Phi(\abs{x}) = \Phi(x)$ for all $x$). The converse is a theorem of von Neumann. ↩︎