What

Given a matrix $$A$$, its QR decomposition is to find matrices $$Q$$ and $$R$$ such that

\begin{aligned} A &= [x_1\ x_2\ \cdots\ x_m] \\ Q &= [\epsilon_1\ \epsilon_2\ \cdots\ \epsilon_m] \\ R &= \left[\begin{matrix} r_{11} & r_{12} & \cdots & r_{1m} \\ 0 & r_{22} & \cdots & r_{2m} \\ \vdots & & \ddots & \\ 0 & 0 & \cdots & r_{mm} \end{matrix}\right] \\ A &= QR \\ Q^TQ &= I \\ \end{aligned}

i.e., $$Q$$ is an orthogonal matrix and $$R$$ is a upper-triangular matrix.

How

Gramm-Schmidt orthogonalization is a method to perform QR decomposition: Consider matrix $$A$$ of $$m$$ columns, as denoted above. We first normalize the first column $$x_1$$ into unit length, $$\epsilon_1 = \frac{1}{r_{11}}x_1 \textrm{, where } r_{11}=||x_1||.$$ Then decompose the second column $$x_2$$ into components parallel and orthogonal to column vector $$\epsilon_1$$,

\begin{aligned} r_{12} &= \epsilon_1^T x_2 \\ r_{22} &= ||x_2 - r_{12}\epsilon_1|| \\ \epsilon_2 &= \frac{1}{r_{22}}(x_2 - r_{12}\epsilon_1) \\ x_2 &= r_{12}\epsilon_1 + (x_2 - r_{12}\epsilon_1) = r_{12}\epsilon_1 + r_{22}\epsilon_2 \end{aligned}

$$r_{12}$$ is measuring the length of $$x_2$$ projected to the unit vector $$\epsilon_1$$. Thus $$x_2 - r_{12}\epsilon_1$$ is a vector orthogonal to $$x_1$$, which the unit vector of such is represented by $$\epsilon_2$$.

Subsequently, we decompose column vector $$x_k$$ into components parallel to $$x_1,x_2,\cdots,x_{k-1}$$ and a component that is orthogonal to all of them. For example, on $$x_3$$, we have

\begin{aligned} r_{13} &= \epsilon_1^T x_3 \\ r_{23} &= \epsilon_2^T x_3 \\ r_{33} &= ||x_3 - r_{13}\epsilon_1 - r_{23}\epsilon_2|| \\ \epsilon_3 &= \frac{1}{r_{33}}(x_3 - r_{13}\epsilon_1 - r_{23}\epsilon_2) \\ x_3 &= r_{13}\epsilon_1 + r_{23}\epsilon_2 + (x_3 - r_{13}\epsilon_1 - r_{23}\epsilon_2) = r_{13}\epsilon_1 + r_{23}\epsilon_2 + r_{33}\epsilon_3 \end{aligned}

Note, matrices $$A$$ and $$Q$$ are of the same dimension, and may not be a square shape, but matrix $$R$$ is always a square matrix.

Why

Assume we have a system of linear equations $$Ax=b$$, for which $$x$$ is the vector of unknown and $$b$$ is a vector of constants. If we decompose $$A=QR$$, we have

\begin{aligned} QRx &= b \\ Q^TQRx &= Q^Tb \\ Rx &= Q^Tb \end{aligned}

Now since $$R$$ is a upper-triangular matrix, we can use Gaussian elimination to find $$x$$.