Mathematics for Artifical Intelligence : Linear Algebra

A simplified guide on how to prep up on Mathematics for Artificial Intelligence, Machine Learning and Data Science: Linear Algebra (Important Pointers only)

Module - I : Linear Algebra

I. Vector and its properties.

A mathematical entity with magnitude and direction denoted as $v$ or $\overrightarrow{\mathrm{<b>v</b>}}$

$\stackrel{}{In\; component\; form\; represented\; as\; ; }$

$\mathbf{v} = \begin{pmatrix} v_1 \\ v_2 \end{pmatrix}$$\stackrel{}{}$

$\stackrel{}{In\; Cartesian\; coordinates\; represented\; as\; :}$

$\mathbf{v} = v_1 \mathbf{i} + v_2 \mathbf{j} + v_3 \mathbf{k}$ 

 Types of Vectors : Zero Vector (magnitude ->0, no direction) , Unit Vector (magnitude ->1, directional), Position Vector (position of a point relative to its origin) , Equal Vectors (same magnitude and same direction) , Opposite Vectors (same magnitude but opposite direction).

Vector operations:

1. Addition and Subtraction

• Vector Addition: $$\mathbf{u}+\mathbf{v}=\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right)+\left(\begin{array}{c}{v}_1\\ v_2\end{array}\right)=\left(\begin{array}{c}{u}_1+v_1\\ u_2+v_2\end{array}\right)$$
• Vector Subtraction: $$\mathbf{u}-\mathbf{v}=\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right)-\left(\begin{array}{c}{v}_1\\ v_2\end{array}\right)=\left(\begin{array}{c}{u}_1-v_1\\ u_2-v_2\end{array}\right)$$

2. Scalar Multiplication

• Multiplying a vector by a scalar $$k\mathbf{v}=k\left(\begin{array}{c}{v}_1\\ v_2\end{array}\right)=\left(\begin{array}{c}k{v}_1\\ k{v}_2\end{array}\right)$$

3. Dot Product (Scalar Product)

• The dot product of two vectors is a scalar.$$\mathbf{u}\cdot \mathbf{v}=u_1{v}_1+u_2{v}_2+\cdots +u_n{v}_n$$
• The dot product can also be expressed in terms of the magnitudes of the vectors and the angle between them: $$\mathbf{u}\cdot \mathbf{v}=\mathrm{\parallel }\mathbf{u}\mathrm{\parallel }\mathrm{\parallel }\mathbf{v}\mathrm{\parallel }\cos \theta$$

4. Cross Product (Vector Product)

The cross product is only defined in three-dimensional space.$$\mathbf{u}\times \mathbf{v}=\mid \begin{array}{ccc}\mathbf{i}& \mathbf{j}& \mathbf{k}\\ u_1& u_2& u_3\\ v_1& v_2& v_3\end{array}\mid$$

 

Vector Properties:

• Commutativity of Addition: $\mathbf{u}+\mathbf{v}=\mathbf{v}+\mathbf{u}$
• Associativity of Addition: $\mathbf{u}+(\mathbf{v}+\mathbf{w})=(\mathbf{u}+\mathbf{v})+\mathbf{w}$
• Distributivity: $k(\mathbf{u}+\mathbf{v})=k\mathbf{u}+k\mathbf{v}$
• Zero Vector: $\mathbf{v}+\mathbf{0}=\mathbf{v}$
• Negative Vector: $\mathbf{v}+(-\mathbf{v})=\mathbf{0}$

 

II. Vector Spaces and Subspaces.

 A collection of vectors that can be added together and multiplied by scalars (real or complex numbers) and still remain within the set.

Denoted as $V$ over a field $F$ (usually $\mathbb{R}$ or $\mathbb{C}$)

Vector Addition: For vectors $\mathbf{u},\mathbf{v}\in V$ and $\mathbf{w} \in V$,  $\mathbf{u}+\mathbf{v}=\mathbf{w}$

Scalar Multiplication: For $a \in F$ and vectors $\mathbf{v} \in V$ and  $\mathbf{w}\in V$,    $a \mathbf{v} = \mathbf{w}$.

Important Axioms: 

Associativity of Addition: $(\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v} + \mathbf{w})$

Commutativity of Addition: $\mathbf{u}+\mathbf{v}=\mathbf{v}+\mathbf{u}$

Identity Element of Addition: There exists an element $\mathbf{0} \in V$ such that $\mathbf{v} + \mathbf{0} = \mathbf{v}$ for all $\mathbf{v}\in \mathrm{V.}$

Inverse Elements of Addition: For each $\mathbf{v} \in V$, there exists an element $-\mathbf{v} \in V$such that $\mathbf{v}+(-\mathbf{v})=\mathbf{0.}$

Compatibility of Scalar Multiplication with Field Multiplication: $a(b\mathbf{v}) = (ab)\mathbf{v}$ for all $a, b \in F$ and $\mathbf{v}\in V$

Identity Element of Scalar Multiplication: $1\mathbf{v} = \mathbf{v}$for all $\mathbf{v} \in V$.

Distributivity of Scalar Multiplication with Respect to Vector Addition: $a(\mathbf{u} + \mathbf{v}) = a\mathbf{u} + a\mathbf{v}$ for all $a\in F$ and $\mathbf{u}, \mathbf{v} \in V$

Distributivity of Scalar Multiplication with Respect to Field Addition: $(a + b)\mathbf{v} = a\mathbf{v} + b\mathbf{v}$ for all $a, b \in F$ and $\mathbf{v}\in \mathrm{V.}$

Subspaces:

A subset $W$ of a vector space $V$ is called a subspace of $V$ if:

• Zero Vector: $\mathbf{0} \in W$.
• Closed under Addition: For all $\mathbf{u}, \mathbf{v} \in W$, $\mathbf{u}+\mathbf{v}\in W$.
• Closed under Scalar Multiplication: For all $a \in F$ and $\mathbf{v}\in W$, $a\mathbf{v}\in W$.

If $W$ satisfies these conditions, $W$ is a subspace of $$VV.

 Eg: The zero subspace: $\{\mathbf{0}\}$ is a subspace of any vector space.

Properties of Subspaces:

• Intersection: The intersection of two subspaces of $V$ is also a subspace of $V$.
• Sum: The sum of two subspaces $W_1$ and $W_2$, defined as $W_1 + W_2 = \{ \mathbf{u} + \mathbf{v} \mid \mathbf{u} \in W_1, \mathbf{v} \in W_2 \}$, is also a subspace of $V$.
• Span: The span of a set of vectors in $V$ is the smallest subspace of $V$ that contains all those vectors.

Spanning Set:

A set of vectors $S = \{\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_k\}$ in $V$ is said to span $V$ if every vector in $V$ can be written as a linear combination of the vectors in $S$.

Basis:

A basis of a vector space $V$ is a set of vectors in $V$ that is linearly independent and spans $V$.

Dimension:

The dimension of a vector space $V$ is the number of vectors in a basis of $V$. It is a measure of the "size" or "degree of freedom" of the vector space.

 

III. Matrices.  

 A rectangular array of numbers, symbols, or expressions arranged in rows and columns, denoted by $A$. If a matrix has $m$ rows and $n$ columns, it is called an $m \times n$ matrix.

For example, a $2 \times 3$ matrix $A$:

$A = \begin{pmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{pmatrix}$

Types of Matrices:

• Square Matrix: A matrix with the same number of rows and columns ($m=n$).
• Row Matrix: A matrix with a single row ($1\times n$).
• Column Matrix: A matrix with a single column ($m\times 1$).
• Zero Matrix: A matrix in which all elements are zero.
• Identity Matrix: A square matrix with ones on the main diagonal and zeros elsewhere.
• Diagonal Matrix: A square matrix in which all off-diagonal elements are zero.
• Symmetric Matrix: A square matrix that is equal to its transpose ($A=A^T$).
• Skew-Symmetric Matrix: A square matrix that is equal to the negative of its transpose ($A = -A^T$).
• Upper Triangular Matrix: A square matrix in which all elements below the main diagonal are zero.
• Lower Triangular Matrix: A square matrix in which all elements above the main diagonal are zero.

 

Matrix Operations :

1. Addition:

$A + B = \begin{pmatrix} a_{11} + b_{11} & a_{12} + b_{12} \\ a_{21} + b_{21} & a_{22} + b_{22} \end{pmatrix}$

2. Subtraction

$A - B = \begin{pmatrix} a_{11} - b_{11} & a_{12} - b_{12} \\ a_{21} - b_{21} & a_{22} - b_{22} \end{pmatrix}$

3. Scalar Multiplication

$cA = \begin{pmatrix} c \cdot a_{11} & c \cdot a_{12} \\ c \cdot a_{21} & c \cdot a_{22} \end{pmatrix}$

4. Matrix Multiplication

Two matrices $A$ ($m \times n$) and $B$ ($n\times p$) can be multiplied to form an $m\times p$ matrix $C$:

$C=AB$

Where each element $c_{ij}$ is calculated as:

$c_{ij}={\sum }_{k=1}^n{a}_{ik}{b}_{kj}$

5. Transpose

 $A^T$ is formed by swapping rows and columns:

$A^T = \begin{pmatrix} a_{11} & a_{21} \\ a_{12} & a_{22} \end{pmatrix}$

6. Determinant

The determinant is a scalar value. For a $2 \times 2$matrix:

$\text{det}(A)=\mid \begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\mid =a_{11}{a}_{22}-a_{12}{a}_2$

 

IV. Matrix Inversion.

The inverse of a square matrix $A$, denoted $A^{-1}$, is the matrix such that $AA^{-1} = A^{-1}A = I$, where $\mathrm{I }$is the identity matrix. The inverse exists only if $\det(A) \neq 0$.

For a $2\times 2$ matrix:

$A^{-1} = \frac{1}{\det(A)} \begin{pmatrix} a_{22} & -a_{12} \\ -a_{21} & a_{11} \end{pmatrix}$

Conditions for Inversion:

Not all matrices have inverses. A matrix $A$ is invertible (or non-singular) if and only if:

1. $A$ is a square matrix.
2. The determinant of $A$ is non-zero, i.e., $\det (A)\mathrm{\ne }0$.

If these conditions are not met, the matrix is said to be singular or non-invertible.

Properties of Inverse Matrix:

• Uniqueness: If $A$ is invertible, its inverse $A^{-1}$ is unique.
• Product of Inverses: If $A$ and $B$ are invertible matrices of the same dimension, then the product $AB$ is invertible 
  $(AB{)}^{-1}=B^{-1}{A}^{-1}$
  
• Inverse of Transpose: If $A$ is invertible, then
  $(A^T{)}^{-1}=(A^{-1}{)}^T$
• Inverse of a Scalar Multiple: If $A$ is invertible and $c$ is a non-zero scalar, then
  $(cA)^{-1} = \frac{1}{c}A^{-1}$

Methods for Finding the Inverse:

1. Gaussian Elimination

To find the inverse of a matrix $A$ using Gaussian elimination:

1. Form the augmented matrix $[A | I]$, where $I$ is the identity matrix of the same dimension as $A$.
2. Use row operations to transform $[A | I]$into $[I | A^{-1}]$.
3. If this is possible, the matrix $A$ is invertible and the right half of the augmented matrix will be $A^{-1}$.

2. Adjugate Method

For a square matrix $A$, the inverse can also be found using the adjugate (or adjoint) and the determinant:

$A^{-1} = \frac{1}{\det(A)} \text{adj}(A)$

where $\text{adj}(A)$ is the adjugate of $A$. The steps are:

1. Compute the determinant $\det(A)$.
2. Find the matrix of cofactors.
3. Transpose the matrix of cofactors to get the adjugate.
4. Divide each entry of the adjugate by $\det(A)$.

3. Using Elementary Matrices

An elementary matrix is obtained by performing a single elementary row operation on an identity matrix. The inverse of $A$ can be found by expressing $A$ as a product of elementary matrices:

$A = E_1 E_2 \cdots E_k$

Then,

$A^{-1} = E_k^{-1} \cdots E_2^{-1} E_1^{-1}$

Eg:

Let's find the inverse of the following 2x2 matrix $A$:

$A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$

The inverse is given by:

$A^{-1} = \frac{1}{\det(A)} \begin{pmatrix} d & -b \\ -c & a \end{pmatrix}$

where $\det(A) = ad - bc$

For example, if:

$A = \begin{pmatrix} 2 & 3 \\ 1 & 4 \end{pmatrix}$

Then:

$\det(A) = (2)(4) - (3)(1) = 8 - 3 = 5$

$A^{-1} = \frac{1}{5} \begin{pmatrix} 4 & -3 \\ -1 & 2 \end{pmatrix} = \begin{pmatrix} 0.8 & -0.6 \\ -0.2 & 0.4 \end{pmatrix}$

Applications

• Solving Linear Systems: Given $A\mathbf{x}=\mathbf{b}$, if $A$ is invertible, the solution is $\mathbf{x}=A^{-1}\mathbf{b}$.
• Computer Graphics: Inverse matrices are used for transforming coordinates and manipulating images.
• Control Theory: Inverse matrices are essential in system design and stability analysis.

 

V. Properties of Determinants. 

1. Determinant of the Identity Matrix

   The determinant of an identity matrix $I$ of any size is 1:

   $$\det(I) = 1$$

2. Determinant of a Diagonal Matrix

   The determinant of a diagonal matrix (a square matrix in which all off-diagonal elements are zero) $D$ with elements $d_{11},d_{22},\dots ,d_{nn}$:

   $$\det(D) = d_{11} \cdot d_{22} \cdot \ldots \cdot d_{nn$$

3. Determinant of a Triangular Matrix

   Similar to diagonal matrices, the determinant of a triangular matrix (either upper or lower triangular) is the product of its diagonal elements.

4. Determinant of the Transpose

   The determinant of a matrix is equal to the determinant of its transpose:

   $$\det(A) = \det(A^T)$$

5. Multiplicative Property

   The determinant of the product of two matrices is the product of their determinants:

   $$\det(AB) = \det(A) \cdot \det(B)$$

6. Determinant of an Inverse

   If $A$ is an invertible matrix, the determinant of its inverse is the reciprocal of the determinant of $A$:

   $$\det(A^{-1}) = \frac{1}{\det(A$$

7. Determinant of a Scalar Multiple

   If $A$ is an $n \times n$matrix and $c$ is a scalar, the determinant of the scalar multiple of $A$ is:

   $$\det(cA) = c^n \det(A)$$

8. Row and Column Operations

   • Row Interchange: Swapping two rows (or two columns) of a matrix changes the sign of the determinant:

     $$\det(B) = -\det(A)$$

     if $B$ is obtained by interchanging two rows (or columns) of $A$.

   • Row Scaling: Multiplying a row (or column) by a scalar multiplies the determinant by the same scalar:

     $$\det(B) = k \cdot \det(A)$$

     if $B$ is obtained by multiplying a row (or column) of $A$ by $k$.

   • Row Addition: Adding a multiple of one row (or column) to another row (or column) does not change the determinant:

     $$\det(B) = \det(A)$$

     if $B$ is obtained by adding a multiple of one row (or column) to another row (or column) of $A$.

9. Determinant of a Block Matrix

   For a block diagonal matrix:

   $$A = \begin{pmatrix} A_1 & 0 \\ 0 & A_2 \end{pmatrix}$$

   The determinant is the product of the determinants of the blocks:

   $$\det(A) = \det(A_1) \cdot \det(A_2)$$

10. Singular Matrix

    A matrix is singular (non-invertible) if and only if its determinant is zero:

    $$\det(A) = 0 \iff A \text{ is singular}$$

11. Linearity in Rows and Columns

    The determinant is a linear function in each row and each column. 

     
    

VI. Eigen Values and Eigen Vectors.

Given a square matrix $A$ of dimension $n\times n$:

• Eigenvalue: A scalar $\lambda$ is called an eigenvalue of $A$ if there exists a non-zero vector $\mathbf{v}$ (called an eigenvector) such that:

  $$A\mathbf{v}=\lambda \mathbf{v}$$

• Eigenvector: A non-zero vector $\mathbf{v}$ is called an eigenvector of $A$ corresponding to the eigenvalue $\lambda$ if it satisfies the above equation.

 Finding Eigenvalues and Eigenvectors

To find the eigenvalues and eigenvectors of a matrix $A$:

1. Eigenvalues: Solve the characteristic equation:

   $$\det(A - \lambda I) = 0$$

   This equation is derived from the equation $A\mathbf{v}=\lambda \mathbf{v}$ by rearranging it to:

   $$(A - \lambda I) \mathbf{v} = 0$$

   For non-trivial solutions (non-zero $\mathbf{v}$), the matrix $A - \lambda I$must be singular, which means its determinant must be zero.

2. Eigenvectors: For each eigenvalue $\lambda$, solve the linear system:

   $$(A - \lambda I) \mathbf{v} = 0$$

   to find the corresponding eigenvector(s) $\mathbf{v}$.
   

Properties of Eigenvalues and Eigenvectors

1. Sum of Eigenvalues: The sum of the eigenvalues of a matrix $A$ is equal to the trace of $A$ (the sum of the diagonal elements).

   $$\sum_{i=1}^{n} \lambda_i = \text{tr}(A)$$

2. Product of Eigenvalues: The product of the eigenvalues of a matrix $A$ is equal to the determinant of $A$.

   $$\prod_{i=1}^{n} \lambda_i = \det(A)$$

3. Eigenvectors of Different Eigenvalues: Eigenvectors corresponding to distinct eigenvalues are linearly independent.

4. Diagonalizability: A matrix $A$ is diagonalizable if it has $n$ linearly independent eigenvectors. In such cases, $A$ can be written as:

   $$A = PDP^{-1}$$

   where $P$ is the matrix of eigenvectors and $D$ is the diagonal matrix of eigenvalues.

5. Similarity Transformation: If $A$ and $B$ are similar matrices, they have the same eigenvalues.

6. Power of a Matrix: If $A$ is diagonalizable, then $A^k$ can be expressed as:

   $$A^k = PD^kP^{-1}$$

   where $D$ is the diagonal matrix with eigenvalues of $A$ on the diagonal, and $D^k$ is simply raising each diagonal entry to the $k$-th power.

Applications:

1. Differential Equations: Eigenvalues and eigenvectors are used to solve systems of linear differential equations.
2. Stability Analysis: In control theory, the stability of a system can be analyzed using the eigenvalues of the system matrix.
3. Principal Component Analysis (PCA): In statistics, PCA uses eigenvalues and eigenvectors of the covariance matrix to reduce the dimensionality of data.
4. Quantum Mechanics: Eigenvalues and eigenvectors are used to solve the Schrödinger equation.
5. Vibration Analysis: In mechanical engineering, the natural frequencies (eigenvalues) and mode shapes (eigenvectors) of a system are analyzed.

 

VII. Diagonalization of Matrices.  

Diagonalization of matrices is a process by which a given square matrix $A$ is decomposed into a product of three matrices: $$A=PDP−1A = PDP^{-1}, where $P$ is an invertible matrix whose columns are the eigenvectors of $A$, $D$ is a diagonal matrix whose entries are the eigenvalues of $A$, and $P^{-1}$ is the inverse of $P$.

Here’s a step-by-step guide to diagonalizing a matrix:

1. Find the Eigenvalues of $A$:

   • Solve the characteristic equation $\det (A-\lambda I)=0$ for $\lambda$. The solutions ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n$ are the eigenvalues of $A$.

2. Find the Eigenvectors of $A$:

   • For each eigenvalue $\lambda$, solve the equation $(A-\lambda I)\mathbf{v}=0$ for the eigenvector $\mathbf{v}$.

3. Form the Matrix $P$:

   • Construct the matrix $P$ using the eigenvectors as columns.

4. Form the Diagonal Matrix $D$:

   • Construct the matrix $D$ by placing the eigenvalues ${\lambda }_i$ on the diagonal. The order of the eigenvalues in $D$ should correspond to the order of the eigenvectors in $P$.

5. Verify the Diagonalization:

   • Verify that $A=PD{P}^{-\mathrm{1 }}$ by calculating both sides of the equation.
    

VIII. Singular Value Decomposition (SVD).

For a given $m\times$ matrix $A$, the SVD is given by:

$A=U\mathrm{\Sigma }{V}^T$

where:

• $U$ is an $m\times m$ orthogonal matrix whose columns are called the left singular vectors of $A$.
• $\mathrm{\Sigma }$ is an $m\times n$ diagonal matrix with non-negative real numbers on the diagonal, known as the singular values of $A$.
• $V$ is an $n\times n$ orthogonal matrix whose columns are called the right singular vectors of $A$.

Steps to Compute the SVD

1. Compute $A^{T}A$ and $A{A}^T$:

   • These are both symmetric matrices.

2. Find the eigenvalues and eigenvectors:

   • Compute the eigenvalues and eigenvectors of $A^{T}A$. The eigenvalues are the squares of the singular values of $A$.
   • Compute the eigenvalues and eigenvectors of $A{A}^T$. The eigenvectors form the columns of $U$ and $V$.

3. Construct $\mathrm{\Sigma }$:

   • The singular values ${\sigma }_i$ are the square roots of the eigenvalues of $A^{T}A$.

4. Form the matrices $U$ and $V$:

   • The columns of $U$ are the eigenvectors of $A{A}^T$.
   • The columns of $V$ are the eigenvectors of $A^{T}A$.

Applications of SVD in Machine Learning:

1. Reducing Dimensions in Data (in PCA - Principal Component Analysis)

2. Compressing Images

3. Removing Noise

4. Making Recommendation systems (like Netflix or Amazon)

5. Understanding Text in Latent Semantic Analysis (LSA)

6. Designing Control Systems like robots or aircraft

7. Face Recognition

8. Analyzing Genetic Datasets

9. Compressing Data for Communication

10. Preventing Overfitting in ML models.

11. Quantum Computing

 

$$k \mathbf{v} = k \begin{pmatrix} v_1 \\ v_2 \end{pmatrix} = \begin{pmatrix} k v_1 \\ k v_2 \end{pmatri$$