Mathematics II

Linear transformation. A map $T:V\to W$ between vector spaces is linear if for all $\mathbf u,\mathbf v\in V$ and scalars $c$:

1. $T(\mathbf u+\mathbf v)=T(\mathbf u)+T(\mathbf v)$,
2. $T(c\mathbf u)=cT(\mathbf u)$.

Example. $T:\mathbb R^2\to\mathbb R^2$, $T(x,y)=(x+y,\,2x)$. Then $T(\mathbf u+\mathbf v)=(u_1+v_1+u_2+v_2,\ 2u_1+2v_1)=T(\mathbf u)+T(\mathbf v)$ and $T(c\mathbf u)=cT(\mathbf u)$, so $T$ is linear. Every matrix transformation $\mathbf x\mapsto A\mathbf x$ is linear.

(In the given problem $A$ is $3\times2$, so $T(\mathbf x)=A\mathbf x$ maps $\mathbb R^2\to\mathbb R^3$.)

(a) Find $T(\mathbf v)$. $T(\mathbf v)=A\mathbf v=\begin{bmatrix}1&-3\\3&5\\-1&7\end{bmatrix}\begin{bmatrix}2\\-1\end{bmatrix}=\begin{bmatrix}2+3\\6-5\\-2-7\end{bmatrix}=\begin{bmatrix}5\\1\\-9\end{bmatrix}.$

(b) Find $\mathbf x$ with $T(\mathbf x)=\mathbf b$. Solve $A\mathbf x=\mathbf b$:

$$\left[\begin{array}{cc|c}1&-3&3\\3&5&2\\-1&7&4\end{array}\right] \xrightarrow[R_3+R_1]{R_2-3R_1}\left[\begin{array}{cc|c}1&-3&3\\0&14&-7\\0&4&7\end{array}\right].$$

From $R_2$: $x_2=-\tfrac12$. Then $R_3$ requires $4x_2=7$, i.e. $-2=7$, which is false. Equivalently, the row $R_3-\tfrac{4}{14}R_2$ becomes $[0\ 0\mid 9]$.

The system is inconsistent, so there is no $\mathbf x\in\mathbb R^2$ whose image under $T$ is $\mathbf b$; $\mathbf b=(3,2,4)^T$ is not in the range of $T$.

Since $T(\mathbf{x})=\begin{bmatrix}0&-1\\1&0\end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix}=\begin{bmatrix}-x_2\\x_1\end{bmatrix}$ (a rotation through $90^\circ$ counter-clockwise):

$T(u)=T\begin{bmatrix}4\\1\end{bmatrix}=\begin{bmatrix}-1\\4\end{bmatrix},\qquad T(v)=T\begin{bmatrix}2\\3\end{bmatrix}=\begin{bmatrix}-3\\2\end{bmatrix},$

$T(u+v)=T\begin{bmatrix}6\\4\end{bmatrix}=\begin{bmatrix}-4\\6\end{bmatrix}.$

Check of linearity: $T(u)+T(v)=\begin{bmatrix}-1-3\\4+2\end{bmatrix}=\begin{bmatrix}-4\\6\end{bmatrix}=T(u+v)$. ✓

Condition for an inverse. A square matrix $A$ has an inverse if and only if $\det A\neq 0$ (i.e. $A$ is non-singular, of full rank). Then $A^{-1}=\dfrac{1}{\det A}\operatorname{adj}A$.

Determinant. Expanding along the first row:

$\det A=5(0\cdot 8-3\cdot(-3))-1(1\cdot 8-3\cdot 4)+2(1\cdot(-3)-0\cdot 4)=45+4-6=43\neq 0.$

So $A^{-1}$ exists.

Cofactors.

$C_{11}=9,\ C_{12}=-(8-12)=4,\ C_{13}=-3,$ $C_{21}=-(8+6)=-14,\ C_{22}=40-8=32,\ C_{23}=-(-15-4)=19,$ $C_{31}=3-0=3,\ C_{32}=-(15-2)=-13,\ C_{33}=0-1=-1.$

The adjoint is the transpose of the cofactor matrix:

$\operatorname{adj}A=\begin{bmatrix}9&-14&3\\4&32&-13\\-3&19&-1\end{bmatrix}$

$A^{-1}=\frac{1}{43}\begin{bmatrix}9&-14&3\\4&32&-13\\-3&19&-1\end{bmatrix}$

Check. Row 1 of $A$ times the columns of $\operatorname{adj}A$ gives $45+4-6=43$, $-70+32+38=0$, $15-13-2=0$. The other rows give $(0,43,0)$ and $(0,0,43)$ in the same way, so $A\operatorname{adj}A=43I$.

The eigenvalues of $A=\begin{pmatrix}6&3&-8\\0&-2&0\\1&0&-3\end{pmatrix}$ are the roots of $\det(A-\lambda I)=0$.

$\det(A-\lambda I)=\begin{vmatrix}6-\lambda&3&-8\\0&-2-\lambda&0\\1&0&-3-\lambda\end{vmatrix}$

Expanding along the second row, which has only one non-zero entry:

$=(-2-\lambda)\begin{vmatrix}6-\lambda&-8\\1&-3-\lambda\end{vmatrix}=(-2-\lambda)\left[(6-\lambda)(-3-\lambda)+8\right].$

Now $(6-\lambda)(-3-\lambda)+8=\lambda^2-3\lambda-18+8=\lambda^2-3\lambda-10=(\lambda-5)(\lambda+2)$.

So the characteristic equation is

$-(\lambda+2)^2(\lambda-5)=0.$

$\lambda=5,\quad \lambda=-2\ (\text{repeated twice}).$

Check: the trace is $6-2-3=1$ and $5-2-2=1$. ✓

Equality of vectors. Two vectors in $\mathbb R^2$ are equal iff their corresponding entries are equal: $\begin{bmatrix}a\\b\end{bmatrix}=\begin{bmatrix}c\\d\end{bmatrix}\iff a=c\ \text{ and }\ b=d.$ Example: $\begin{bmatrix}2\\-1\end{bmatrix}=\begin{bmatrix}1+1\\-1\end{bmatrix}$, but $\begin{bmatrix}2\\-1\end{bmatrix}\neq\begin{bmatrix}-1\\2\end{bmatrix}$ because order matters.

Computation (the given vectors lie in $\mathbb R^3$; the same entrywise rules apply): $\mathbf u+3\mathbf v=\begin{bmatrix}1\\-3\\2\end{bmatrix}+\begin{bmatrix}3\\-3\\9\end{bmatrix}=\begin{bmatrix}4\\-6\\11\end{bmatrix},$ $\mathbf u-2\mathbf v=\begin{bmatrix}1\\-3\\2\end{bmatrix}-\begin{bmatrix}2\\-2\\6\end{bmatrix}=\begin{bmatrix}-1\\-1\\-4\end{bmatrix}.$

Signals $y_k^{(1)}=1^k,\ y_k^{(2)}=(-2)^k,\ y_k^{(3)}=3^k$ are linearly independent if their Casoratian is nonzero for at least one $k$. The Casoratian is $C(k)=\begin{vmatrix}1^k&(-2)^k&3^k\\1^{k+1}&(-2)^{k+1}&3^{k+1}\\1^{k+2}&(-2)^{k+2}&3^{k+2}\end{vmatrix}.$

At $k=0$: $C(0)=\begin{vmatrix}1&1&1\\1&-2&3\\1&4&9\end{vmatrix}=1(-18-12)-1(9-3)+1(4+2)=-30-6+6=-30.$

In general, factoring $1^k,(-2)^k,3^k$ out of the columns gives $C(k)=(1)^k(-2)^k3^k\,(-30)=-30\,(-6)^k\neq0$ for every $k$.

Since the Casoratian is nonzero, the three signals are linearly independent.

Setting. Take $\mathbf u,\mathbf v\neq\mathbf 0$ in the plane. The segment is the one joining $-\mathbf v$ and $\mathbf v$ (symmetric about the origin), and $L$ is the line through $\mathbf 0$ and $\mathbf u$, i.e. $L=\{t\mathbf u: t\in\mathbb R\}$. The perpendicular bisector of a segment is the set of points equidistant from its two endpoints. We show $\mathbf u\perp\mathbf v$ iff $L$ is the perpendicular bisector of this segment.

Perpendicular bisector. A point $\mathbf x$ is equidistant from $-\mathbf v$ and $\mathbf v$ iff $\|\mathbf x-\mathbf v\|^2=\|\mathbf x+\mathbf v\|^2\iff \|\mathbf x\|^2-2\mathbf x\cdot\mathbf v+\|\mathbf v\|^2=\|\mathbf x\|^2+2\mathbf x\cdot\mathbf v+\|\mathbf v\|^2\iff \mathbf x\cdot\mathbf v=0.$ So the perpendicular bisector is $B=\{\mathbf x:\mathbf x\cdot\mathbf v=0\}$. It passes through the midpoint $\mathbf 0$ and is perpendicular to the segment (direction $\mathbf v$).

($\Rightarrow$) If $\mathbf u\cdot\mathbf v=0$, then every point $t\mathbf u$ satisfies $(t\mathbf u)\cdot\mathbf v=0$, so $L\subseteq B$. In the plane $B$ is a line, and $L$ is a line, so $L=B$.

($\Leftarrow$) If $L=B$, then $\mathbf u\in L=B$, so $\mathbf u\cdot\mathbf v=0$, i.e. $\mathbf u\perp\mathbf v$.

Equivalently, $\mathbf u\perp\mathbf v\iff\|\mathbf u+\mathbf v\|=\|\mathbf u-\mathbf v\|$, which says $\mathbf u$ is equidistant from $-\mathbf v$ and $\mathbf v$. $\blacksquare$

Consistent and inconsistent systems. A system of linear equations $A\mathbf{x}=\mathbf{b}$ is consistent if it has at least one solution (exactly one or infinitely many). By the rank test this happens when $\operatorname{rank}(A)=\operatorname{rank}[A\,|\,\mathbf{b}]$. It is inconsistent if it has no solution, i.e. $\operatorname{rank}(A)<\operatorname{rank}[A\,|\,\mathbf{b}]$ (the echelon form contains a row $[0\ 0\ \dots\ 0\,|\,c]$ with $c\neq 0$).

• Consistent example: $x+y=2,\ x-y=0$ has the unique solution $x=y=1$.
• Inconsistent example: $x+y=2,\ x+y=3$ has no solution.

Test and solve. The augmented matrix is

$[A\,|\,\mathbf{b}]=\begin{bmatrix}1&1&1&4\\1&2&2&2\\2&2&1&5\end{bmatrix}$

Apply $R_2\to R_2-R_1$ and $R_3\to R_3-2R_1$:

$\begin{bmatrix}1&1&1&4\\0&1&1&-2\\0&0&-1&-3\end{bmatrix}$

This is in echelon form with three non-zero rows, so $\operatorname{rank}(A)=\operatorname{rank}[A\,|\,\mathbf{b}]=3=$ number of unknowns. The system is consistent with a unique solution.

Back substitution:

$-z=-3\Rightarrow z=3,\qquad y+z=-2\Rightarrow y=-5,\qquad x+y+z=4\Rightarrow x=6.$

Check: $6-5+3=4$, $6-10+6=2$, $12-10+3=5$. All three equations hold.

$x=6,\quad y=-5,\quad z=3.$

The least-squares solution $\hat{\mathbf{x}}$ of $A\mathbf{x}=\mathbf{b}$ solves the normal equations $A^TA\hat{\mathbf{x}}=A^T\mathbf{b}$.

Let the columns of $A$ be $c_1=(1,1,1,1)$, $c_2=(3,1,1,3)$, $c_3=(5,0,2,3)$.

$A^TA=\begin{bmatrix}4&8&10\\8&20&26\\10&26&38\end{bmatrix},\qquad A^T\mathbf{b}=\begin{bmatrix}3+5+7+3\\9+5+7+9\\15+0+14+9\end{bmatrix}=\begin{bmatrix}18\\30\\38\end{bmatrix}$

The normal equations (each divided by 2) are

$2x_1+4x_2+5x_3=9,\qquad 4x_1+10x_2+13x_3=15,\qquad 5x_1+13x_2+19x_3=19.$

Eliminate $x_1$:

• Equation 2 minus 2 times equation 1: $2x_2+3x_3=-3$.
• 2 times equation 3 minus 5 times equation 1: $6x_2+13x_3=-7$.

Subtracting 3 times the first of these from the second gives $4x_3=2$, so $x_3=\tfrac12$. Then $x_2=\dfrac{-3-\frac32}{2}=-\dfrac94$, and

$x_1=\frac{9-4x_2-5x_3}{2}=\frac{9+9-\frac52}{2}=\frac{31}{4}.$

Check in equation 3: $5\cdot\frac{31}{4}+13\cdot\left(-\frac94\right)+19\cdot\frac12=\frac{155-117+38}{4}=19$. ✓

$\hat{\mathbf{x}}=\left(\frac{31}{4},\ -\frac{9}{4},\ \frac12\right)^T.$

Here $A=\begin{pmatrix}-1&-2\\5&9\end{pmatrix}$ and $B=\begin{pmatrix}9&2\\k&-1\end{pmatrix}$.

$AB=\begin{pmatrix}-9-2k&-2+2\\45+9k&10-9\end{pmatrix}=\begin{pmatrix}-9-2k&0\\45+9k&1\end{pmatrix}$

$BA=\begin{pmatrix}-9+10&-18+18\\-k-5&-2k-9\end{pmatrix}=\begin{pmatrix}1&0\\-k-5&-2k-9\end{pmatrix}$

Equating entries of $AB$ and $BA$:

• $(1,1)$: $-9-2k=1\Rightarrow k=-5$.
• $(2,1)$: $45+9k=-k-5\Rightarrow 10k=-50\Rightarrow k=-5$.
• $(2,2)$: $1=-2k-9\Rightarrow k=-5$.

All conditions agree, so $AB=BA$ exactly when

$k=-5.$

Consistent and inconsistent systems. A system of linear equations is consistent if it has at least one solution (unique or infinitely many), which happens when $\operatorname{rank}A=\operatorname{rank}[A\,|\,\mathbf{b}]$. It is inconsistent if it has no solution, which happens when $\operatorname{rank}A<\operatorname{rank}[A\,|\,\mathbf{b}]$.

• Consistent example: $x+y=3,\ x-y=1$ with solution $x=2,\ y=1$.
• Inconsistent example: $x+y=1,\ 2x+2y=5$, which has no solution.

Test and solve. The system is $x-2y=5$, $-x+y+5z=2$, $y+z=0$. The augmented matrix is

$\begin{bmatrix}1&-2&0&5\\-1&1&5&2\\0&1&1&0\end{bmatrix}$

$R_2\to R_2+R_1$:

$\begin{bmatrix}1&-2&0&5\\0&-1&5&7\\0&1&1&0\end{bmatrix}$

$R_3\to R_3+R_2$:

$\begin{bmatrix}1&-2&0&5\\0&-1&5&7\\0&0&6&7\end{bmatrix}$

There are three non-zero rows, so $\operatorname{rank}A=\operatorname{rank}[A\,|\,\mathbf{b}]=3$. The system is consistent with a unique solution.

Back substitution:

$6z=7\Rightarrow z=\frac76,\qquad -y+5z=7\Rightarrow y=5z-7=-\frac76,\qquad x=5+2y=5-\frac73=\frac83.$

Check: $y+z=0$ ✓; $-\frac83-\frac76+\frac{35}{6}=-\frac{16}{6}+\frac{28}{6}=2$ ✓.

$x=\frac83,\quad y=-\frac76,\quad z=\frac76.$

The least-squares solution satisfies the normal equations $A^TA\hat{\mathbf{x}}=A^T\mathbf{b}$.

With $A=\begin{bmatrix}1&-6\\1&-2\\1&1\\1&7\end{bmatrix}$ and $\mathbf{b}=\begin{bmatrix}-1\\2\\1\\6\end{bmatrix}$:

$A^TA=\begin{bmatrix}4&-6-2+1+7\\0&36+4+1+49\end{bmatrix}=\begin{bmatrix}4&0\\0&90\end{bmatrix}$

$A^T\mathbf{b}=\begin{bmatrix}-1+2+1+6\\6-4+1+42\end{bmatrix}=\begin{bmatrix}8\\45\end{bmatrix}$

The normal equations are $4\hat{x}_1=8$ and $90\hat{x}_2=45$, giving

$\hat{x}_1=2,\qquad \hat{x}_2=\frac12.$

$\hat{\mathbf{x}}=\begin{bmatrix}2\\ \tfrac12\end{bmatrix},$

i.e. the best-fit line $y=2+\tfrac12x$.

With $A=\begin{bmatrix}2&5\\-3&1\end{bmatrix}$ and $B=\begin{bmatrix}4&-5\\3&k\end{bmatrix}$:

$AB=\begin{bmatrix}8+15&-10+5k\\-12+3&15+k\end{bmatrix}=\begin{bmatrix}23&5k-10\\-9&15+k\end{bmatrix}$

$BA=\begin{bmatrix}8+15&20-5\\6-3k&15+k\end{bmatrix}=\begin{bmatrix}23&15\\6-3k&15+k\end{bmatrix}$

Equating entries:

• $(1,2)$: $5k-10=15\Rightarrow k=5$.
• $(2,1)$: $-9=6-3k\Rightarrow k=5$.

Both agree, so $AB=BA$ for

$k=5.$

Change-of-coordinate matrix. The matrix $P_{C\leftarrow B}$ has as its columns the $C$-coordinate vectors of the vectors in $B$, so that $[x]_C=P_{C\leftarrow B}[x]_B$.

From $b_1=-c_1+4c_2$ and $b_2=5c_1-3c_2$:

$[b_1]_C=\begin{bmatrix}-1\\4\end{bmatrix},\qquad [b_2]_C=\begin{bmatrix}5\\-3\end{bmatrix},$

$P_{C\leftarrow B}=\begin{bmatrix}-1&5\\4&-3\end{bmatrix}.$

Coordinates of $x=5b_1+3b_2$. Here $[x]_B=\begin{bmatrix}5\\3\end{bmatrix}$, so

$[x]_C=\begin{bmatrix}-1&5\\4&-3\end{bmatrix}\begin{bmatrix}5\\3\end{bmatrix}=\begin{bmatrix}-5+15\\20-9\end{bmatrix}=\begin{bmatrix}10\\11\end{bmatrix}.$

Check directly: $5b_1+3b_2=5(-c_1+4c_2)+3(5c_1-3c_2)=10c_1+11c_2$. ✓

The columns of $A$ are linearly independent iff $A\mathbf x=\mathbf 0$ has only the trivial solution, equivalently (for a square matrix) iff $\det A\neq0$.

Row reduction. $A=\begin{bmatrix}0&1&4\\1&2&-1\\5&8&0\end{bmatrix}\xrightarrow{R_1\leftrightarrow R_2}\begin{bmatrix}1&2&-1\\0&1&4\\5&8&0\end{bmatrix}\xrightarrow{R_3-5R_1}\begin{bmatrix}1&2&-1\\0&1&4\\0&-2&5\end{bmatrix}\xrightarrow{R_3+2R_2}\begin{bmatrix}1&2&-1\\0&1&4\\0&0&13\end{bmatrix}.$ There is a pivot in every column, so there are no free variables and $A\mathbf x=\mathbf 0$ has only $\mathbf x=\mathbf 0$.

Determinant check. $\det A=0(0+8)-1(0+5)+4(8-10)=-5-8=-13\neq0.$

Hence the columns of $A$ are linearly independent.

Null space. The null space of an $m\times n$ matrix $A$ is the set of all solutions of the homogeneous equation $A\mathbf x=\mathbf 0$: $\operatorname{Nul}A=\{\mathbf x\in\mathbb R^n : A\mathbf x=\mathbf 0\}.$ It is a subspace of $\mathbb R^n$.

Test for $\mathbf v$. $\mathbf v\in\operatorname{Nul}A$ iff $A\mathbf v=\mathbf 0$: $A\mathbf v=\begin{bmatrix}-1&-3&2\\-5&-9&1\end{bmatrix}\begin{bmatrix}5\\-3\\-2\end{bmatrix}=\begin{bmatrix}-5+9-4\\-25+27-2\end{bmatrix}=\begin{bmatrix}0\\0\end{bmatrix}.$ Since $A\mathbf v=\mathbf 0$, the vector $\mathbf v$ is in $\operatorname{Nul}A$.

The unit vector in the direction of $\mathbf u$ is $\hat{\mathbf u}=\dfrac{\mathbf u}{\|\mathbf u\|}$.

$\|\mathbf u\|=\sqrt{1^2+(-2)^2+2^2+3^2}=\sqrt{1+4+4+9}=\sqrt{18}=3\sqrt2.$

$\mathbf v=\frac1{3\sqrt2}(1,-2,2,3)=\left(\frac1{3\sqrt2},\ -\frac2{3\sqrt2},\ \frac2{3\sqrt2},\ \frac3{3\sqrt2}\right)=\left(\frac{\sqrt2}{6},\ -\frac{\sqrt2}{3},\ \frac{\sqrt2}{3},\ \frac{\sqrt2}{2}\right).$

Check. $\|\mathbf v\|^2=\frac{1+4+4+9}{18}=1$ ✓.

Model $y=a_0+a_1x$ with data $(0,1),(1,1),(1,1),(2,2),(3,2)$. The design matrix and observation vector are $X=\begin{bmatrix}1&0\\1&1\\1&1\\1&2\\1&3\end{bmatrix},\qquad \mathbf y=\begin{bmatrix}1\\1\\1\\2\\2\end{bmatrix}.$

Normal equations $X^TX\mathbf a=X^T\mathbf y$ with $n=5$:

• $\sum x=0+1+1+2+3=7$
• $\sum y=1+1+1+2+2=7$
• $\sum x^2=0+1+1+4+9=15$
• $\sum xy=0+1+1+4+6=12$

$\begin{bmatrix}5&7\\7&15\end{bmatrix}\begin{bmatrix}a_0\\a_1\end{bmatrix}=\begin{bmatrix}7\\12\end{bmatrix}.$

$\det=75-49=26$. Using Cramer's rule, $a_0=\frac{7\cdot15-7\cdot12}{26}=\frac{21}{26},\qquad a_1=\frac{5\cdot12-7\cdot7}{26}=\frac{11}{26}.$

Check. $5\cdot\frac{21}{26}+7\cdot\frac{11}{26}=\frac{182}{26}=7$ ✓; $7\cdot\frac{21}{26}+15\cdot\frac{11}{26}=\frac{312}{26}=12$ ✓.

$y=\frac{21}{26}+\frac{11}{26}x\approx0.808+0.423x.$

The columns of $A=\begin{bmatrix}-2&8&-1\\0&0&0\\0&-5&3\end{bmatrix}$ are linearly independent iff $A\mathbf x=\mathbf 0$ has only the trivial solution.

Reduction. Swap $R_2\leftrightarrow R_3$: $\begin{bmatrix}-2&8&-1\\0&-5&3\\0&0&0\end{bmatrix}.$ Only two pivots (columns 1 and 2) exist, so $x_3$ is a free variable.

Solving $A\mathbf x=\mathbf 0$: $-5x_2+3x_3=0\Rightarrow x_2=\tfrac35x_3$, and $-2x_1+8x_2-x_3=0\Rightarrow x_1=\tfrac{19}{10}x_3$. Taking $x_3=10$ gives the nontrivial solution $\mathbf x=(19,6,10)^T$: $19\begin{bmatrix}-2\\0\\0\end{bmatrix}+6\begin{bmatrix}8\\0\\-5\end{bmatrix}+10\begin{bmatrix}-1\\0\\3\end{bmatrix}=\begin{bmatrix}-38+48-10\\0\\-30+30\end{bmatrix}=\mathbf 0.$ (Also $\det A=0$ because of the zero row.)

Hence the columns of $A$ are linearly dependent.

Null space. For an $m\times n$ matrix $A$, the null space $\text{Nul}\,A$ is the set of all vectors $x\in\mathbb{R}^n$ that satisfy the homogeneous equation $Ax=0$:

$\text{Nul}\,A=\{x\in\mathbb{R}^n : Ax=\mathbf{0}\}$

It is a subspace of $\mathbb{R}^n$.

Showing $v$ is in Nul $A$. Compute $Av$:

$Av=\begin{bmatrix}-1&-3&2\\-5&-9&1\end{bmatrix}\begin{bmatrix}5\\-3\\-2\end{bmatrix}=\begin{bmatrix}(-1)(5)+(-3)(-3)+2(-2)\\(-5)(5)+(-9)(-3)+1(-2)\end{bmatrix}=\begin{bmatrix}-5+9-4\\-25+27-2\end{bmatrix}=\begin{bmatrix}0\\0\end{bmatrix}$

Since $Av=\mathbf{0}$, the vector $v$ belongs to the null space of $A$.