Linear systems – direct methods | Nicholas Hu

The LU and PLU factorizations

Wed, 20 Dec 2023 00:00:00 +0000

$$ \newcommand{\set}[1]{\{ #1 \}} \newcommand{\Set}[1]{\left \{ #1 \right\}} \renewcommand{\emptyset}{\varnothing} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} \newcommand{\R}{\mathbb{R}} \newcommand{\Rn}{\mathbb{R}^n} \newcommand{\Rm}{\mathbb{R}^m} \newcommand{\C}{\mathbb{C}} \newcommand{\F}{\mathbb{F}} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\Abs}[1]{\left\lvert #1 \right\rvert} \newcommand{\inner}[2]{\langle #1, #2 \rangle} \newcommand{\Inner}[2]{\left\langle #1, #2 \right\rangle} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\Norm}[1]{\left\lVert #1 \right\rVert} \newcommand{\trans}{{\top}} \newcommand{\span}{\mathop{\mathrm{span}}} \definecolor{cblue}{RGB}{31, 119, 180} \definecolor{corange}{RGB}{255, 127, 14} \definecolor{cgreen}{RGB}{44, 160, 44} \definecolor{cred}{RGB}{214, 39, 40} \definecolor{cpurple}{RGB}{148, 103, 189} \definecolor{cbrown}{RGB}{140, 86, 75} \definecolor{cpink}{RGB}{227, 119, 194} \definecolor{cgrey}{RGB}{127, 127, 127} \definecolor{cyellow}{RGB}{188, 189, 34} \definecolor{cteal}{RGB}{23, 190, 207} $$

The LU factorization

Let $\F$ be a field and $A \in \F^{n \times n}$. Given a $b \in \F^n$, recall that the linear system $Ax = b$ can be solved by performing Gaussian elimination (also known as row reduction) on the augmented matrix $\begin{bmatrix} A & b \end{bmatrix}$ to transform it into an augmented matrix of the form $\begin{bmatrix} U & y \end{bmatrix}$, where $U \in \F^{n \times n}$ is upper triangular and $y \in \F^n$, which represents an equivalent linear system $Ux = y$. Such a system can then be solved readily by backward substitution. In fact, this process (under certain conditions) amounts to computing an LU factorization of $A$ as $LU$, where $L \in \F^{n \times n}$ is unit lower triangular and $U \in \F^{n \times n}$ is upper triangular.

To illustrate this, suppose that $\F = \R$ and $$ \underbrace{\begin{bmatrix} 1 & 5 & 1 & 2 \\ 2 & 6 & 3 & 4 \\ 3 & 7 & 2 & 4 \\ 4 & 8 & 1 & 3 \end{bmatrix}}_{A} \underbrace{\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \end{bmatrix}}_{x\vphantom{A}} = \underbrace{\begin{bmatrix} 7 \\ 8 \\ 7 \\ 8 \end{bmatrix}}_{b\vphantom{A}}. $$ Then the variable $x_1$ can be eliminated from the second equation $2 x_1 + 6 x_2 + 3 x_3 + 4 x_4 = 8$ by multiplying the first equation $1 x_1 + 5 x_2 + 1 x_3 + 2 x_4 = 7$ by $\frac{2}{1}$ and subtracting the resulting equation from the second. The quotient $\frac{2}{1} = 2$ is called a multiplier and the divisor $1$ is called a pivot. In other words (ignoring the vector $b$ for now), $$ \begin{bmatrix} 2 & 6 & 3 & 4 \end{bmatrix} - 2 \, {\color{cblue} \begin{bmatrix} 1 & 5 & 1 & 2 \end{bmatrix}} = {\color{corange} \begin{bmatrix} 0 & -4 & 1 & 0 \end{bmatrix}}\,. $$

Similarly, $x_1$ can be eliminated from the third and fourth equations, yielding the factorization

$$ \begin{align*} \begin{bmatrix} 1 & 5 & 1 & 2 \\ 2 & 6 & 3 & 4 \\ 3 & 7 & 2 & 4 \\ 4 & 8 & 1 & 3 \end{bmatrix} &= \begin{bmatrix} 1 \, {\color{cblue} \begin{bmatrix} 1 & 5 & 1 & 2 \end{bmatrix} \hphantom{{} + 1 \begin{bmatrix} 0 & -8\hphantom{2} & -1 & -2 \end{bmatrix}}} \\ 2 \, {\color{cblue} \begin{bmatrix} 1 & 5 & 1 & 2 \end{bmatrix}} + 1 \, {\color{corange} \begin{bmatrix} 0 & -4\hphantom{2} & \hphantom{-}1 & \hphantom{-}0 \end{bmatrix}} \\ 3 \, {\color{cblue} \begin{bmatrix} 1 & 5 & 1 & 2 \end{bmatrix}} + 1 \begin{bmatrix} 0 & -8\hphantom{2} & -1 & -2 \end{bmatrix} \\ 4 \, {\color{cblue} \begin{bmatrix} 1 & 5 & 1 & 2 \end{bmatrix}} + 1 \begin{bmatrix} 0 & -12 & -3 & -5 \end{bmatrix} \end{bmatrix} \\ &= \begin{bmatrix} 1 & \\ 2 & 1 & \\ 3 & & 1 & \\ 4 & & & 1 \end{bmatrix} \begin{bmatrix} \color{cblue} 1 & \color{cblue} 5 & \color{cblue} 1 & \color{cblue} 2 \\ & \color{corange} -4 & \color{corange} 1 & \color{corange} 0 \\ & -8 & -1 & -2 \\ & -12 & -3 & -5 \end{bmatrix} \end{align*} $$

(where blanks denote zeroes).

The variable $x_2$ can then be eliminated analogously from the new third and fourth equations. For instance,

$$ \begin{bmatrix} 0 & -8 & -1 & -2 \end{bmatrix} - 2 \, {\color{corange} \begin{bmatrix} 0 & -4 & 1 & 0 \end{bmatrix}} = {\color{cgreen} \begin{bmatrix} 0 & 0 & -3 & -2 \end{bmatrix}}\,. $$

In view of the previous step, this means that

$$ \begin{bmatrix} 3 & 7 & 2 & 4 \end{bmatrix} - 3 \, {\color{cblue} \begin{bmatrix} 1 & 5 & 1 & 2 \end{bmatrix}} - 2 \, {\color{corange} \begin{bmatrix} 0 & -4 & 1 & 0 \end{bmatrix}} = {\color{cgreen} \begin{bmatrix} 0 & 0 & -3 & -2 \end{bmatrix}}\,. $$

Thus, after the second step of the factorization, we have

$$ \begin{bmatrix} 1 & 5 & 1 & 2 \\ 2 & 6 & 3 & 4 \\ 3 & 7 & 2 & 4 \\ 4 & 8 & 1 & 3 \end{bmatrix} = \begin{bmatrix} 1 & \\ 2 & 1 & \\ 3 & 2 & 1 & \\ 4 & 3 & & 1 \end{bmatrix} \begin{bmatrix} \color{cblue} 1 & \color{cblue} 5 & \color{cblue} 1 & \color{cblue} 2 \\ & \color{corange} -4 & \color{corange} 1 & \color{corange} 0 \\ & & \color{cgreen} -3 & \color{cgreen} -2 \\ & & -6 & -5 \end{bmatrix}. $$

Finally, an entirely analogous third step completes the LU factorization:

$$ \underbrace{\begin{bmatrix} 1 & 5 & 1 & 2 \\ 2 & 6 & 3 & 4 \\ 3 & 7 & 2 & 4 \\ 4 & 8 & 1 & 3 \end{bmatrix}}_{A} = \underbrace{\begin{bmatrix} 1 & \\ 2 & 1 & \\ 3 & 2 & 1 & \\ 4 & 3 & 2 & 1 \end{bmatrix}}_{L} \underbrace{\begin{bmatrix} \color{cblue} 1 & \color{cblue} 5 & \color{cblue} 1 & \color{cblue} 2 \\ & \color{corange} -4 & \color{corange} 1 & \color{corange} 0 \\ & & \color{cgreen} -3 & \color{cgreen} -2 \\ & & & \color{cred} -1 \end{bmatrix}}_{U}. $$

In general, we can regard LU factorization as partitioning a matrix $A \in \F^{n \times n}$ as $$ A = \begin{bmatrix} \alpha & b^\trans \\ c & D \end{bmatrix}, $$ where $\alpha \in \F$, $b \in \F^{n-1}$, $c \in \F^{n-1}$, and $D \in \F^{(n-1) \times (n-1)}$. Provided that the pivot $\alpha$ is nonzero, we can eliminate $c$ by using the multipliers $\frac{c}{\alpha}$ to write $\begin{bmatrix} c & D \end{bmatrix} - \frac{c}{\alpha} \begin{bmatrix} \alpha & b^\trans \end{bmatrix} = \begin{bmatrix} 0 & A' \end{bmatrix}$ for some $A’ \in \F^{(n-1) \times (n-1)}$; the matrix $A' = D - \frac{c}{\alpha}b^\trans$ is known as the Schur complement of $\begin{bmatrix} \alpha \end{bmatrix}$ in $A$. This yields the factorization $$ A = \begin{bmatrix} 1 & \\ \frac{c}{\alpha} & I \end{bmatrix} \begin{bmatrix} \alpha & b^\trans \\ & A' \end{bmatrix}, $$ which can be recursively continued by computing an LU factorization $L' U'$ of $A’$. If one exists, then $\begin{bmatrix} c & D \end{bmatrix} - \frac{c}{\alpha} \begin{bmatrix} \alpha & b^\trans \end{bmatrix} - L' \begin{bmatrix} 0 & U' \end{bmatrix} = \begin{bmatrix} 0 & 0 \end{bmatrix}$, so $$ A = \underbrace{\begin{bmatrix} 1 & \\ \frac{c}{\alpha} & L' \end{bmatrix}}_{L} \underbrace{\begin{bmatrix} \alpha & b^\trans \\ & U' \end{bmatrix}}_{U} $$ is an LU factorization of $A$.

Existence and uniqueness

Let $a_{ij}^{(k)}$ denote the ( $i$, $j$)-entry of $A$ after the $k$^th step in the LU factorization. From the example above, it is clear that $A$ will have an LU factorization provided that the pivots $a_{11}^{(0)}, a_{22}^{(1)}, \dots, a_{n-1,n-1}^{(n-2)}$ are nonzero. Moreover, we see after $k-1$ steps that the $k$^th leading principal minor of $A$ is equal to $a_{11}^{(0)} a_{22}^{(1)} \cdots a_{kk}^{(k-1)}$, so $A$ will have an LU factorization if its first $n-1$ leading principal minors are nonzero.

If $A$ is nonsingular, this condition is also necessary because the $k$^th leading principal minor of $A$ must be equal to $u_{11} u_{22} \cdots u_{kk}$. Furthermore, the LU factorization is unique in this case, for if $L_1 U_1$ and $L_2 U_2$ are both LU factorizations of $A$, then $L_2^{-1} L_1 = U_2 U_1^{-1}$ must be simultaneously unit lower triangular and upper triangular, and hence equal to the identity.

Solving linear systems

If the same row operations that were applied to $A$ are applied to $b$ as well, evidently the resulting vector $y$ will satisfy $b = Ly$ (since $\begin{bmatrix} A & b \end{bmatrix}$ = $L \begin{bmatrix} U & y \end{bmatrix}$). Therefore the solutions of $Ax = b$ will indeed be those of $Ux = y$. Notably, once an LU factorization of $A$ has been computed, it is possible to solve $Ax = b$ for any given $b \in \F^n$ by solving $Ly = b$ for $y$ and then $Ux = y$ for $x$, reusing the computed factors $L$ and $U$! (The former system can be solved readily by forward substitution and the latter by backward substitution.)

The PLU factorization

It is possible to produce an LU-like factorization for any $A \in \F^{n \times n}$ by allowing for row interchanges in addition to the elementary row operation above. The resulting PLU factorization consists of a permutation matrix $P \in \F^{n \times n}$ along with matrices $L$ and $U$ as above such that $PA = LU$ (or equivalently, $A = P^\trans LU$).

As an illustration, consider the matrix $$ \underbrace{\begin{bmatrix} 4 & 4 & 4 & 4 \\ 1 & 1 & 2 & 2 \\ 4 & 5 & 5 & 6 \\ 2 & 4 & 4 & 4 \end{bmatrix}}_{A}. $$ Following one elimination step as before, we arrive at $$ A = \begin{bmatrix} 4 & 4 & 4 & 4 \\ \vphantom{\frac{1}{1}} 1 & 1 & 2 & 2 \\ 4 & 5 & 5 & 6 \\ \vphantom{\frac{1}{1}} 2 & 4 & 4 & 4 \end{bmatrix} = \begin{bmatrix} 1 & \\ \frac{1}{4} & 1 & \\ 1 & & 1 & \\ \frac{1}{2} & & & 1 \end{bmatrix} \begin{bmatrix} \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 \\ \vphantom{\frac{1}{1}} & \color{corange} 0 & \color{corange} 1 & \color{corange} 1 \\ & 1 & 1 & 2 \\ \vphantom{\frac{1}{1}} & \color{cpurple} 2 & \color{cpurple} 2 & \color{cpurple} 2 \end{bmatrix}. $$

Since the (2, 2)-entry of the factor on the right is zero, it is impossible to introduce zeroes in the second column using this entry as a pivot. However, by interchanging the second row with a row below it whose entry in the second column is nonzero, we can obtain a nonzero pivot to perform the elimination.¹ Also, note that had such a row not existed, it could only have been because all subdiagonal entries in the second column were zero, in which case we could have immediately continued to the next elimination step!

For example, suppose we wish to use the (4, 2)-entry, $2$, as a pivot. We have

$$ \begin{cases} \begin{bmatrix} 1 & 1 & 2 & 2 \end{bmatrix} = \frac{1}{4} \, {\color{cblue} \begin{bmatrix} 4 & 4 & 4 & 4 \end{bmatrix}} + {\color{corange} \begin{bmatrix} 0 & 0 & 1 & 1 \end{bmatrix}} \\ \begin{bmatrix} 2 & 4 & 4 & 4 \end{bmatrix} = \frac{1}{2} \, {\color{cblue} \begin{bmatrix} 4 & 4 & 4 & 4 \end{bmatrix}} + {\color{cpurple} \begin{bmatrix} 0 & 2 & 2 & 2 \end{bmatrix}} \end{cases}\,, $$

so if $P_2$ is the permutation matrix that interchanges the second and fourth rows, then

$$ P_2 A = \begin{bmatrix} 4 & 4 & 4 & 4 \\ \vphantom{\frac{1}{1}} 2 & 4 & 4 & 4 \\ 4 & 5 & 5 & 6 \\ \vphantom{\frac{1}{1}} 1 & 1 & 2 & 2 \\ \end{bmatrix} = \begin{bmatrix} 1 & \\ \frac{1}{2} & 1 & \\ 1 & & 1 & \\ \frac{1}{4} & & & 1 \end{bmatrix} \begin{bmatrix} \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 \\ \vphantom{\frac{1}{1}} & \color{cpurple} 2 & \color{cpurple} 2 & \color{cpurple} 2 \\ & 1 & 1 & 2 \\ \vphantom{\frac{1}{1}} & \color{corange} 0 & \color{corange} 1 & \color{corange} 1 \\ \end{bmatrix}, $$

from which elimination can proceed as in the LU factorization. Namely,

$$ P_2 A = \begin{bmatrix} 4 & 4 & 4 & 4 \\ \vphantom{\frac{1}{1}} 2 & 4 & 4 & 4 \\ \vphantom{\frac{1}{1}} 4 & 5 & 5 & 6 \\ \vphantom{\frac{1}{1}} 1 & 1 & 2 & 2 \\ \end{bmatrix} = \begin{bmatrix} 1 & \\ \frac{1}{2} & 1 & \\ 1 & \frac{1}{2} & 1 & \\ \frac{1}{4} & 0 & & 1 \end{bmatrix} \begin{bmatrix} \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 \\ \vphantom{\frac{1}{1}} & \color{cpurple} 2 & \color{cpurple} 2 & \color{cpurple} 2 \\ \vphantom{\frac{1}{1}} & & \color{cgreen} 0 & \color{cgreen} 1 \\ \vphantom{\frac{1}{1}} & & \color{cbrown} 1 & \color{cbrown} 1 \\ \end{bmatrix}. $$

Once again, we see that a row interchange is required. We have

$$ \begin{cases} \begin{bmatrix} 4 & 5 & 5 & 6 \end{bmatrix} = 1 \, {\color{cblue} \begin{bmatrix} 4 & 4 & 4 & 4 \end{bmatrix}} + \frac{1}{2} \, {\color{cpurple} \begin{bmatrix} 0 & 2 & 2 & 2 \end{bmatrix}} + {\color{cgreen} \begin{bmatrix} 0 & 0 & 0 & 1 \end{bmatrix}} \\ \begin{bmatrix} 1 & 1 & 2 & 2 \end{bmatrix} = \frac{1}{4} \, {\color{cblue} \begin{bmatrix} 4 & 4 & 4 & 4 \end{bmatrix}} + 0 \, {\color{cpurple} \begin{bmatrix} 0 & 2 & 2 & 2 \end{bmatrix}} + {\color{cbrown} \begin{bmatrix} 0 & 0 & 1 & 1 \end{bmatrix}} \\ \end{cases}\,, $$

so if $P_3$ is the permutation matrix that interchanges the third and fourth rows, then

$$ P_3 P_2 A = \begin{bmatrix} 4 & 4 & 4 & 4 \\ \vphantom{\frac{1}{1}} 2 & 4 & 4 & 4 \\ \vphantom{\frac{1}{1}} 1 & 1 & 2 & 2 \\ \vphantom{\frac{1}{1}} 4 & 5 & 5 & 6 \\ \end{bmatrix} = \begin{bmatrix} 1 & \\ \frac{1}{2} & 1 & \\ \frac{1}{4} & 0 & 1 & \\ 1 & \frac{1}{2} & & 1 \end{bmatrix} \begin{bmatrix} \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 \\ \vphantom{\frac{1}{1}} & \color{cpurple} 2 & \color{cpurple} 2 & \color{cpurple} 2 \\ \vphantom{\frac{1}{1}} & & \color{cbrown} 1 & \color{cbrown} 1 \\ \vphantom{\frac{1}{1}} & & \color{cgreen} 0 & \color{cgreen} 1 \\ \end{bmatrix}. $$

The product of all the permutation matrices used for row interchanges becomes the permutation matrix $P$; if no interchange is performed in the $k$^th step, we can regard this as multiplication by $P_k = I$ at that step. For instance, after the final elimination step in our example (which is trivial since the (4, 3)-entry is already zero), $$ \underbrace{P_3 P_2 P_1}_{P} A = \underbrace{\begin{bmatrix} 4 & 4 & 4 & 4 \\ \vphantom{\frac{1}{1}} 2 & 4 & 4 & 4 \\ \vphantom{\frac{1}{1}} 1 & 1 & 2 & 2 \\ \vphantom{\frac{1}{1}} 4 & 5 & 5 & 6 \\ \end{bmatrix}}_{PA} = \underbrace{\begin{bmatrix} 1 & \\ \frac{1}{2} & 1 & \\ \frac{1}{4} & 0 & 1 & \\ 1 & \frac{1}{2} & 0 & 1 \end{bmatrix}}_{L} \underbrace{\begin{bmatrix} \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 & \color{cblue} 4 \\ \vphantom{\frac{1}{1}} & \color{cpurple} 2 & \color{cpurple} 2 & \color{cpurple} 2 \\ \vphantom{\frac{1}{1}} & & \color{cbrown} 1 & \color{cbrown} 1 \\ \vphantom{\frac{1}{1}} & & & \color{cpink} 1 \\ \end{bmatrix}}_{U}. $$

In general, given a matrix $A \in \F^{n \times n}$, either there exists a permutation matrix $P_1 \in \F^{n \times n}$ such that $P_1 A$ is of the form $$ P_1 A = \begin{bmatrix} \alpha & b^\trans \\ c & D \end{bmatrix} $$ for some nonzero $\alpha \in \F$, or the first column of $A$ is zero, in which case $P_1 A$ is of the form $$ P_1 A = \begin{bmatrix} \alpha & b^\trans \\ 0 & D \end{bmatrix} $$ for any permutation matrix $P_1 \in \F^{n \times n}$. Let $\hat{c} = \frac{c}{\alpha}$ in the former case and $\hat{c} = 0$ in the latter. Then $$ P_1 A = \begin{bmatrix} 1 & \\ \hat{c} & I \end{bmatrix} \begin{bmatrix} \alpha & b^\trans \\ & A' \end{bmatrix}, $$ where $A’ = D - \hat{c} b^\trans$, and if $P' A' = L’ U’$ is a PLU factorization of $A’$, then $$ \underbrace{\begin{bmatrix} 1 & \\ & P' \end{bmatrix} P_1}_{P} A = \underbrace{\begin{bmatrix} 1 & \\ P' \hat{c} & L' \end{bmatrix}}_{L} \underbrace{\begin{bmatrix} \alpha & b^\trans \\ & U' \end{bmatrix}}_{U} $$ is a PLU factorization of $A$.

Solving linear systems

By the same reasoning as above, row-reducing $b$ in the manner that $A$ was reduced results in a vector $y$ satisfying $Pb = Ly$. Therefore the solutions of $Ax = b$ will be those of $Ux = y$. Once a PLU factorization of $A$ has been computed, it is possible to solve $Ax = b$ for any given $b \in \F^n$ by solving $Ly = Pb$ for $y$ and then $Ux = y$ for $x$, reusing the computed factors $L$ and $U$ and the permutation represented by $P$.

In practice, implementations of PLU factorization typically perform a row interchange that maximizes the absolute value of the pivot, regardless of whether it is needed to prevent division by zero. With this convention, all multipliers are at most $1$ in absolute value. ↩︎

Symmetric factorizations

Tue, 17 Jun 2025 00:00:00 +0000

PDF

$$ %% Sets and functions %% \newcommand{\set}[1]{\{ #1 \}} \newcommand{\Set}[1]{\left \{ #1 \right\}} \renewcommand{\emptyset}{\varnothing} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} \newcommand{\R}{\mathbb{R}} \newcommand{\Rn}{\mathbb{R}^n} \newcommand{\Rm}{\mathbb{R}^m} \newcommand{\C}{\mathbb{C}} \newcommand{\F}{\mathbb{F}} %% Linear algebra %% \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\Abs}[1]{\left\lvert #1 \right\rvert} \newcommand{\inner}[2]{\langle #1, #2 \rangle} \newcommand{\Inner}[2]{\left\langle #1, #2 \right\rangle} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\Norm}[1]{\left\lVert #1 \right\rVert} \newcommand{\tp}{{\top}} \newcommand{\trans}{{\top}} \newcommand{\span}{\operatorname{span}} \newcommand{\im}{\operatorname{im}} \newcommand{\ker}{\operatorname{ker}} \newcommand{\rank}{\operatorname{rank}} \newcommand{\proj}{\operatorname{proj}} \newcommand{\proj}[1]{\mathop{\mathrm{proj}_{#1}}} \newcommand{\refl}{\operatorname{refl}} \newcommand{\refl}[1]{\mathop{\mathrm{refl}_{#1}}} \newcommand{\K}{\mathcal{K}} \newcommand{\L}{\mathcal{L}} \renewcommand{\epsilon}{\varepsilon} \newcommand{\conj}{\overline} \newcommand{\sign}{\operatorname{sign}} %% Colours %% \definecolor{cblue}{RGB}{31, 119, 180} \definecolor{corange}{RGB}{255, 127, 14} \definecolor{cgreen}{RGB}{44, 160, 44} \definecolor{cred}{RGB}{214, 39, 40} \definecolor{cpurple}{RGB}{148, 103, 189} \definecolor{cbrown}{RGB}{140, 86, 75} \definecolor{cpink}{RGB}{227, 119, 194} \definecolor{cgrey}{RGB}{127, 127, 127} \definecolor{cyellow}{RGB}{188, 189, 34} \definecolor{cteal}{RGB}{23, 190, 207} $$

The LDL* factorization

Let $A \in \C^{n \times n}$ and suppose that its first $n-1$ leading principal minors are nonzero. Then $A$ has a unique LU factorization $A = LU$, and moreover $u_{11}, \dots, u_{n-1,\,n-1}$ are nonzero (being the pivots in the LU factorization). As a result, $A$ has a unique LDU factorization $A = LD\hat{U}$, where $L \in \C^{n \times n}$ is unit lower triangular, $D \in \C^{n \times n}$ is diagonal, and $\hat{U} \in \C^{n \times n}$ is unit upper triangular; given by the unique factorization of $U$ as $U = D \hat{U}$. When $A$ is also self-adjoint, we must have $\hat{U} = L^*$, in which case this factorization is called the LDL* factorization (or LDL^T factorization if $A$ is real).

Given a self-adjoint $A \in \C^{n \times n}$, we can also derive the LDL* factorization directly: if $$ A = \begin{bmatrix} \alpha & c^* \\ c & B \end{bmatrix} $$ for some $\alpha \in \C$ (in fact, $\alpha \in \R$ since $A$ is self-adjoint), $c \in \C^{n-1}$, and $B \in \C^{(n-1) \times (n-1)}$, then $$ A = \begin{bmatrix} 1 & \vphantom{\frac{c^*}{\alpha}} \\ \frac{c}{\alpha} & I \end{bmatrix} \begin{bmatrix} \alpha & \vphantom{\frac{c^*}{\alpha}} \\ \vphantom{\frac{c}{\alpha}} & B - \frac{cc^*}{\alpha} \end{bmatrix} \begin{bmatrix} 1 & \frac{c^*}{\alpha} \\ \vphantom{\frac{c}{\alpha}} & I \end{bmatrix}, $$ and if $L’ D’ (L’)^*$ is the LDL* factorization of the Schur complement $A’ = B - \frac{cc^*}{\alpha}$, then $$ A = \underbrace{\begin{bmatrix} 1 & \vphantom{\frac{c^*}{\alpha}} \\ \frac{c}{\alpha} & L' \end{bmatrix}}_L \underbrace{\begin{bmatrix} \alpha & \vphantom{\frac{c^*}{\alpha}} \\ \vphantom{\frac{c}{\alpha}} & D' \end{bmatrix}}_D \underbrace{\begin{bmatrix} 1 & \frac{c^*}{\alpha} \\ \vphantom{\frac{c}{\alpha}} & (L')^* \end{bmatrix}}_{L^*} $$ is the LDL* factorization of $A$.

Diagonal pivoting methods

Let $A \in \C^{n \times n}$ be self-adjoint. Just as an LU factorization may fail to exist because of a zero pivot, an LDL* factorization may fail to exist as well. To obtain a nonzero pivot while preserving the symmetry of $A$, we can interchange two rows of $A$ along with the corresponding columns (illustrated below), which amounts to replacing $A$ with $P_1 A P_1^*$ for some permutation matrix $P_1$. $$ \begin{bmatrix} & 1 & \\ 1 & & \\ & & 1 \end{bmatrix} \begin{bmatrix} 0 & a_{12} & a_{13} \\ a_{21} & {\color{cblue} a_{22}} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \begin{bmatrix} & 1 & \\ 1 & & \\ & & 1 \end{bmatrix} = \begin{bmatrix} {\color{cblue} a_{22}} & a_{21} & a_{23} \\ a_{12} & 0 & a_{13} \\ a_{32} & a_{31} & a_{33} \end{bmatrix} $$ However, using a symmetric interchange as such, pivots can only be selected from the diagonal entries of $A$, which could all be zero despite the matrix itself being nonzero. To remedy this, we can use a permutation to move a nonzero off-diagonal entry at position $(i, j)$ to position $(2, 1)$ (illustrated below), which allows us to then perform a $2 \times 2$ block elimination step. $$ \begin{bmatrix} & 1 & \\ & & 1 \\ 1 & & \end{bmatrix} \begin{bmatrix} 0 & a_{12} & a_{13} \\ a_{21} & 0 & a_{23} \\ a_{31} & {\color{cblue} a_{32}} & 0 \end{bmatrix} \begin{bmatrix} & & 1 \\ 1 & & \\ & 1 & \end{bmatrix} = \begin{bmatrix} 0 & a_{23} & a_{21} \\ {\color{cblue} a_{32}} & 0 & a_{31} \\ a_{12} & a_{13} & 0 \end{bmatrix} $$ In general, if $A$ is nonzero, there exists a permutation matrix $P_1$ such that $$ P_1 A P_1^* = \begin{bmatrix} E & C^* \\ C & B \end{bmatrix}, $$ where the pivot $E \in \C^{s \times s}$ is invertible and $s \in \set{1, 2}$, $C \in \C^{(n-s) \times s}$, and $B \in \C^{(n-s) \times (n-s)}$. We then have $$ P_1 A P_1^* = \begin{bmatrix} I & \\ CE^{-1} & I \end{bmatrix} \begin{bmatrix} E & \\ & B - CE^{-1}C^* \end{bmatrix} \begin{bmatrix} I & E^{-1} C^* \\ & I \end{bmatrix}. $$ Continuing symmetric elimination with the Schur complement $A’ = B - CE^{-1}C^*$, we ultimately produce a factorization of the form $PAP^* = LDL^*$, where $P$ is a permutation matrix, $L$ is unit lower triangular, and $D$ is self-adjoint quasi-diagonal (block diagonal with $1 \times 1$ or $2 \times 2$ blocks).

The Bunch–Parlett factorization

Let $\mu_0 := \max_{i,j} \, \abs{a_{ij}} = \norm{A}_{1,\infty}$ and $\mu_1 := \max_{i} \, \abs{a_{ii}}$. The Bunch–Parlett factorization is a diagonal pivoting method that uses a $1 \times 1$ pivot with $\abs{e_{11}} = \mu_1$ whenever $\mu_1 \geq \alpha \mu_0$ and a $2 \times 2$ pivot with $\abs{e_{21}} = \mu_0$ otherwise, where $\alpha \in (0, 1)$ is a constant chosen to minimize an upper bound for $\mu_0' := \max_{i,j} \, \abs{a_{ij}'} = \norm{A'}_{1, \infty}$.

Namely, for a $1 \times 1$ pivot, we have $$ \begin{align*} \mu_0' &\leq \norm{B}_{1,\infty} + \norm{C}_{1,\infty} \norm{E^{-1}}_{\infty,1} \norm{C^*}_{1,\infty} \\ &\leq \mu_0 + \mu_0 \cdot \frac{1}{\mu_1} \cdot \mu_0 \\ &\leq \left(1 + \frac{1}{\alpha}\right) \mu_0, \end{align*} $$ and for a $2 \times 2$ pivot, we have $$ \begin{align*} \mu_0' &\leq \norm{B}_{1,\infty} + \norm{C}_{1,\infty} \norm{E^{-1}}_{\infty,1} \norm{C^*}_{1,\infty} \\ &\leq \mu_0 + \mu_0 \cdot \frac{2(\mu_0 + \mu_1)}{\abs{\det(E)}} \cdot \mu_0 \\ &\leq \mu_0 + \mu_0 \cdot \frac{2(\mu_0 + \mu_1)}{\mu_0^2 - \mu_1^2} \cdot \mu_0 \\ &< \left(1 + \frac{2}{1-\alpha} \right) \mu_0. \end{align*} $$ To choose $\alpha$, we equate the growth factor $(1 + \frac{1}{\alpha})^2$ for two $1 \times 1$ pivots to the growth factor $1 + \frac{2}{1-\alpha}$ for one $2 \times 2$ pivot, which yields $\alpha = \frac{1 + \sqrt{17}}{8} \approx 0.640$.

The Cholesky factorization

Let $A \in \C^{n \times n}$ be (self-adjoint) positive definite. Then $A$ has a unique LDL* factorization $A = LDL^*$ since its principal submatrices are also positive definite, and moreover $D = (L^{-1}) A (L^{-1})^*$ is positive definite. As a result, $A$ has a unique Cholesky factorization $A = \tilde{L} \tilde{L}^*$, where $\tilde{L} \in \C^{n \times n}$ is lower triangular with positive diagonal entries, given by $\tilde{L} = L \sqrt{D}$.

Given a positive definite $A \in \C^{n \times n}$, we can also derive the Cholesky factorization directly: if $$ A = \begin{bmatrix} \alpha & c^* \\ c & B \end{bmatrix} $$ for some $\alpha \in \C$ (in fact, $\alpha \in \R_{> 0}$ since $A$ is positive definite), $c \in \C^{n-1}$, and $B \in \C^{(n-1) \times (n-1)}$, then $$ A = \begin{bmatrix} \sqrt{\alpha} & \vphantom{\frac{c^*}{\sqrt{\alpha}}} \\ \frac{c}{\sqrt{\alpha}} & I \end{bmatrix} \begin{bmatrix} 1 & \vphantom{\frac{c^*}{\sqrt{\alpha}}} \\ \vphantom{\frac{c}{\sqrt{\alpha}}} & B - \frac{cc^*}{\alpha} \end{bmatrix} \begin{bmatrix} \sqrt{\alpha} & \frac{c^*}{\sqrt{\alpha}} \\ \vphantom{\frac{c}{\sqrt{\alpha}}} & I \end{bmatrix}, $$ and if $\tilde{L}' (\tilde{L}')^*$ is the Cholesky factorization of the Schur complement $A’ = B - \frac{cc^*}{\alpha}$, then $$ A = \underbrace{\begin{bmatrix} \sqrt{\alpha} & \vphantom{\frac{c^*}{\sqrt{\alpha}}} \\ \frac{c}{\sqrt{\alpha}} & \tilde{L}' \end{bmatrix}}_{\tilde{L}} \underbrace{\begin{bmatrix} \sqrt{\alpha} & \frac{c^*}{\sqrt{\alpha}} \\ \vphantom{\frac{c}{\sqrt{\alpha}}} & (\tilde{L}')^* \end{bmatrix}}_{\tilde{L}^*} $$ is the Cholesky factorization of $A$.