Back

Vector Norms, Frobenius Norm, and Determinants

Part 1: Vector Norms + Frobenius Norm + Determinants

Vector Norms

Definition

A norm is a mathematical function used to measure the size, length, or magnitude of a vector or matrix.

The norm of a vector is represented as:

∣∣x∣∣||x||

A norm gives a single numerical value that represents how large a vector is.

Norms are widely used in:

  • Machine Learning
  • Data Science
  • Optimization
  • Computer Vision
  • Numerical Computing

Common types of norms:

  1. L1 Norm (Manhattan Norm)
  2. L2 Norm (Euclidean Norm)
  3. Frobenius Norm (Matrix Norm)

Key Points

1. L1 Norm (Manhattan Norm)

The L1 norm calculates the sum of the absolute values of all vector elements.

Formula:

∣∣x∣∣1=∣x1∣+∣x2∣+...+∣xn∣||x||_1 = |x_1|+|x_2|+...+|x_n|

It is called the Manhattan norm because it measures distance like moving through city blocks.

Example

Given:

X=[−142]X= \begin{bmatrix} -1 \\ 4 \\ 2 \end{bmatrix}

Find the L1 norm.

Solution:

∣∣X∣∣1=∣−1∣+∣4∣+∣2∣||X||_1 = |-1|+|4|+|2| =1+4+2=1+4+2 =7=7

Therefore:

∣∣X∣∣1=7\boxed{||X||_1=7}

2. L2 Norm (Euclidean Norm)

Definition

The L2 norm measures the shortest distance between a vector and the origin.

It is the normal length of a vector in geometry.

Formula:

∣∣x∣∣2=x12+x22+...+xn2||x||_2= \sqrt{x_1^2+x_2^2+...+x_n^2}

The L2 norm is the most commonly used norm in mathematics and machine learning.

Example

Given:

X=[−142]X= \begin{bmatrix} -1 \\ 4 \\ 2 \end{bmatrix}

Calculate the L2 norm.

Solution:

∣∣X∣∣2=(−1)2+42+22||X||_2= \sqrt{(-1)^2+4^2+2^2} =1+16+4=\sqrt{1+16+4} =21=\sqrt{21}

Therefore:

∣∣X∣∣2=21\boxed{||X||_2=\sqrt{21}}

Frobenius Norm

Definition

The Frobenius norm is the extension of the L2 norm from vectors to matrices.

It measures the magnitude of a matrix by treating all matrix elements as vector elements.

Formula:

∣∣A∣∣∗F=∑∗i=1m∑j=1n∣aij∣2||A||*F= \sqrt{ \sum*{i=1}^{m} \sum_{j=1}^{n} |a_{ij}|^2 }

In simple words:

  1. Square every element of the matrix.
  2. Add all squared values.
  3. Take the square root.

Key Points

For a vector:

x=[12]x= \begin{bmatrix} 1\\ 2 \end{bmatrix}

The L2 norm is:

∣∣x∣∣2=12+22||x||_2=\sqrt{1^2+2^2}

For a matrix:

A=[a11a12a21a22]A= \begin{bmatrix} a_{11}&a_{12}\\ a_{21}&a_{22} \end{bmatrix}

The Frobenius norm is:

∣∣A∣∣∗F=a∗112+a122+a212+a222||A||*F= \sqrt{ a*{11}^2+a_{12}^2+a_{21}^2+a_{22}^2 }

The Frobenius norm is basically the matrix version of the L2 norm.

Example / Code

Given:

A=[120010003]A= \begin{bmatrix} 1&2&0\\ 0&1&0\\ 0&0&3 \end{bmatrix}

Find the Frobenius norm.

Solution:

∣∣A∣∣F=12+22+02+02+12+02+02+02+32||A||_F= \sqrt{ 1^2+2^2+0^2+0^2+1^2+0^2+0^2+0^2+3^2 } 1+4+1+9 \sqrt{1+4+1+9} 15 \sqrt{15}

Explanation

The Frobenius norm gives a measurement of the total magnitude of a matrix.

A larger Frobenius norm means:

  • Larger matrix values
  • Greater matrix magnitude

A smaller Frobenius norm means:

  • Smaller values
  • Less magnitude

Applications:

  • Measuring error between matrices
  • Machine learning optimization
  • Matrix approximation
  • Image processing

Output

For:

A=[120010003] A= \begin{bmatrix} 1&2&0\\ 0&1&0\\ 0&0&3 \end{bmatrix}

The Frobenius norm is:

∣∣A∣∣_F=15 \boxed{||A||\_F=\sqrt{15}}

Determinants

Definition

The determinant is a scalar value calculated from a square matrix.

The determinant provides important information about:

  • Matrix invertibility
  • Linear independence
  • Solutions of linear systems

The determinant is represented as:

∣A∣ |A|

Key Points

Properties of Determinants

If:

∣A∣=0 |A|=0

then:

  1. The matrix is singular.
  2. The inverse of the matrix does not exist.
  3. The system does not have a unique solution.
  4. The columns of the matrix are linearly dependent.

If:

∣A∣≠0 |A|\neq0

then:

  1. The matrix is invertible.
  2. The inverse exists.
  3. The columns are linearly independent.
  4. The system has a unique solution.

Determinant of a 2×2 Matrix

For:

A=[abcd] A= \begin{bmatrix} a&b\\ c&d \end{bmatrix}

The determinant is:

∣A∣=ad−bc |A|=ad-bc

Example

Given:

A=[2345] A= \begin{bmatrix} 2&3\\ 4&5 \end{bmatrix}

Calculate the determinant.

Solution:

∣A∣=(2)(5)−(3)(4) |A|=(2)(5)-(3)(4) =10−12 =10-12 =−2 =-2

Therefore:

∣A∣=−2 \boxed{|A|=-2}

Determinant of a 3×3 Matrix

For:

A=[a∗11a∗12a∗13a∗21a∗22a∗23a∗31a∗32a_33] A= \begin{bmatrix} a*{11}&a*{12}&a*{13}\\ a*{21}&a*{22}&a*{23}\\ a*{31}&a*{32}&a\_{33} \end{bmatrix}

The determinant is calculated using expansion:

∣A∣=a∗11∣a∗22a∗23a∗32a_33∣−−−a∗12∣a∗21a∗23a∗31a_33∣−a∗13∣a∗21a∗22a∗31a_32∣ |A|= a*{11} \begin{vmatrix} a*{22}&a*{23}\\ a*{32}&a\_{33} \end{vmatrix} --- a*{12} \begin{vmatrix} a*{21}&a*{23}\\ a*{31}&a\_{33} \end{vmatrix} - a*{13} \begin{vmatrix} a*{21}&a*{22}\\ a*{31}&a\_{32} \end{vmatrix}

Example

Find the determinant:

A=[2−3120−1145] A= \begin{bmatrix} 2&-3&1\\ 2&0&-1\\ 1&4&5 \end{bmatrix}

Using expansion:

∣A∣=2∣0−145∣+(−3)∣2−115∣−1∣2014∣|A|= 2 \begin{vmatrix} 0&-1 \\ 4&5 \end{vmatrix} + (-3) \begin{vmatrix} 2&-1\\ 1&5 \end{vmatrix} - 1 \begin{vmatrix} 2&0\\ 1&4 \end{vmatrix}

First minor:

∣0−145∣=============(0)(5)−(−1)(4)=4\begin{vmatrix} 0&-1\\ 4&5 \end{vmatrix} ============= (0)(5)-(-1)(4)=4

Second minor:

∣2−115∣=============(2)(5)−(−1)(1)=11\begin{vmatrix} 2&-1\\ 1&5 \end{vmatrix} ============= (2)(5)-(-1)(1)=11

Third minor:

∣2014∣=============(2)(4)−(0)(1)=8\begin{vmatrix} 2&0\\ 1&4 \end{vmatrix} ============= (2)(4)-(0)(1)=8

Therefore:

∣A∣=2(4)+3(11)+1(8)|A|=2(4)+3(11)+1(8) =8+33+8=8+33+8 ∣A∣=49\boxed{|A|=49}

Common Mistakes

  • Using determinants for non-square matrices.
  • Forgetting the negative sign in the second term of a 3×3 determinant.
  • Confusing determinant with absolute value.
  • Forgetting that zero determinant means no inverse exists.
  • Mixing L1 and L2 norm formulas.
  • Forgetting to square values before adding in Frobenius norm.

Short Exam Notes

  • Norm measures the magnitude of vectors or matrices.
  • L1 norm:
∣∣x∣∣1=∑∣xi∣||x||_1=\sum |x_i|
  • L2 norm:
∣∣x∣∣2=∑xi2||x||_2=\sqrt{\sum x_i^2}
  • Frobenius norm is the matrix version of L2 norm.
  • Determinant is defined only for square matrices.
  • If:
∣A∣=0|A|=0

the matrix is singular.

  • If:
∣A∣≠0|A|\neq0

the matrix is invertible.

  • Determinants are used later for finding eigenvalues.

Eigenvalues and Eigenvectors Concepts

Part 2: Eigenvalues and Eigenvectors Concepts

Definition

Eigenvalues and eigenvectors are special concepts in linear algebra that describe how a matrix transforms vectors.

When a matrix multiplies a vector, the vector usually changes both its direction and magnitude.

However, there are some special vectors that maintain their direction after transformation. These vectors are called eigenvectors.

The value that describes how much the eigenvector is scaled is called the eigenvalue.

The relationship between a matrix, eigenvector, and eigenvalue is:

Av=λvAv=\lambda v

Where:

  • (A) = square matrix
  • (v) = eigenvector
  • (\lambda) = eigenvalue
  • (Av) = transformed vector

Key Points

1. Understanding Eigenvectors

An eigenvector is a non-zero vector that does not change its direction when multiplied by a matrix.

The matrix only changes the size of the vector.

Mathematically:

Av=λvAv=\lambda v

means:

  • The output of the matrix multiplication is still the same vector direction.
  • The vector is only multiplied by a scalar value.
  • That scalar value is the eigenvalue.

Example / Code

Given matrix:

A=[2112]A= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix}

Consider the vector:

v=[11]v= \begin{bmatrix} 1\\ 1 \end{bmatrix}

Multiply matrix (A) by vector (v):

Av=[2112][11]Av= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix} \begin{bmatrix} 1\\ 1 \end{bmatrix}

Calculate each element:

Av=[2(1)+1(1)1(1)+2(1)]Av= \begin{bmatrix} 2(1)+1(1)\\ 1(1)+2(1) \end{bmatrix} Av=[33]Av= \begin{bmatrix} 3\\ 3 \end{bmatrix}

Rewrite the result:

Av=3[11]Av= 3 \begin{bmatrix} 1\\ 1 \end{bmatrix}

Therefore:

Av=3vAv=3v

The vector:

v=[11]v= \begin{bmatrix} 1\\ 1 \end{bmatrix}

is an eigenvector.

The eigenvalue is:

λ=3\lambda=3

Explanation

Normally, matrix transformations can:

  • Rotate vectors
  • Stretch vectors
  • Shrink vectors
  • Change directions

Example:

A normal vector:

xx

after transformation:

AxAx

may point in a completely different direction.

But an eigenvector behaves differently:

Av=λvAv=\lambda v

The direction remains unchanged.

Only the magnitude changes.

2. Understanding Eigenvalues

Definition

An eigenvalue is a scalar value that represents the scaling factor of an eigenvector.

It tells us how much the vector grows, shrinks, or changes direction.

Effect of Eigenvalues

Positive Eigenvalue

If:

λ>0\lambda>0

The vector keeps the same direction.

Example:

λ=2\lambda=2

The vector becomes twice as large.

Eigenvalue Between 0 and 1

If:

0<λ<10<\lambda<1

The vector becomes smaller.

Example:

λ=0.5\lambda=0.5

The vector becomes half its original size.

Negative Eigenvalue

If:

λ<0\lambda<0

The vector reverses its direction.

Example:

λ=−1\lambda=-1

The vector points in the opposite direction.

Zero Eigenvalue

If:

λ=0\lambda=0

Then:

Av=0Av=0

The vector is transformed into the zero vector.

3. Matrix Transformation Perspective

Definition

Previously, matrices were studied from the vector perspective.

The expression:

AxAx

means:

A matrix transforms a vector.

Now we study matrices from another perspective:

Given a matrix, what special vectors and numbers belong to it?

These special values are:

  • Special vectors → Eigenvectors
  • Special numbers → Eigenvalues

Examples of Matrix Transformations

Zero Transformation

0x=00x=0

A zero matrix transforms every vector into the zero vector.

Identity Transformation

The identity matrix is:

I=[1001]I= \begin{bmatrix} 1&0\\ 0&1 \end{bmatrix}

Multiplying by the identity matrix does not change the vector:

Ix=xIx=x

Scaling Transformation

If:

2I=[2002]2I= \begin{bmatrix} 2&0\\ 0&2 \end{bmatrix}

Then:

2Ix=2x2Ix=2x

The vector becomes twice its size.

Rotation Transformation

A rotation matrix can change the direction of vectors.

A rotation of (180^\circ) gives:

Rπx=−xR_{\pi}x=-x

The vector points in the opposite direction.

4. Importance of Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors are important in many fields:

Machine Learning

Used for:

  • Principal Component Analysis (PCA)
  • Feature extraction
  • Dimensionality reduction

Data Science

Used for:

  • Finding important patterns
  • Reducing large datasets
  • Data compression

Computer Vision

Used for:

  • Image processing
  • Object recognition
  • Feature detection

Engineering and Physics

Used for:

  • Stability analysis
  • Signal processing
  • System modeling

Eigenvalues and Data Compression

Definition

Eigenvalues help identify which eigenvectors contain the most important information in data.

Suppose a dataset has 100 dimensions.

After finding eigenvalues:

(v1,λ1=0.5)(v_1,\lambda_1=0.5) (v2,λ2=0.3)(v_2,\lambda_2=0.3) (v3,λ3=0.2)(v_3,\lambda_3=0.2) (v4,λ4=0.0001)(v_4,\lambda_4=0.0001)

The eigenvalues show the contribution of each eigenvector.

Explanation

Large eigenvalues:

  • Represent important information.
  • Contribute more to the dataset.
  • Should usually be kept.

Small eigenvalues:

  • Represent less important information.
  • Contribute very little.
  • Can often be removed.

This idea is used in dimensionality reduction.

Example 2: Another Eigenvector

Given:

A=[2112]A= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix}

Consider:

v=[−11]v= \begin{bmatrix} -1\\ 1 \end{bmatrix}

Calculate:

Av=[2112][−11]Av= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix} \begin{bmatrix} -1\\ 1 \end{bmatrix}

First row:

2(−1)+1(1)=−12(-1)+1(1)=-1

Second row:

1(−1)+2(1)=11(-1)+2(1)=1

Therefore:

Av=[−11]Av= \begin{bmatrix} -1\\ 1 \end{bmatrix}

This means:

Av=1vAv=1v

Therefore:

λ=1\lambda=1

and:

v=[−11]v= \begin{bmatrix} -1\\ 1 \end{bmatrix}

is an eigenvector.

Common Mistakes

  • Thinking every vector is an eigenvector.
  • Forgetting that eigenvectors must be non-zero vectors.
  • Confusing eigenvalues with eigenvectors.
  • Thinking eigenvalues are vectors.
  • Forgetting that eigenvalues are scalar values.
  • Assuming eigenvectors always keep the same length.
  • Ignoring negative eigenvalues and direction reversal.

Short Exam Notes (Very Concise Revision)

  • Eigenvectors are special vectors whose direction does not change after transformation.
  • Eigenvalues are scaling factors of eigenvectors.
  • The main equation is:
Av=λvAv=\lambda v
  • Eigenvectors must be non-zero.
  • Eigenvalues can be positive, negative, or zero.
  • Large eigenvalues represent important data directions.
  • Eigenvalues and eigenvectors are used in PCA and compression.
  • The eigenvector equation is:
(A−λI)v=0(A-\lambda I)v=0
  • Eigenvalues are found first, then eigenvectors.

Finding Eigenvalues Using Characteristic Equation

Part 3: Finding Eigenvalues (Characteristic Equation)

Definition

The first step in finding eigenvalues is to solve the characteristic equation.

For a square matrix (A), eigenvalues are values of (\lambda) that satisfy:

Av=λvAv=\lambda v

To find the eigenvalues, we rearrange this equation:

Av−λv=0Av-\lambda v=0

Because:

v=Ivv=Iv

we can write:

Av−λIv=0Av-\lambda Iv=0

Taking (v) as a common factor:

(A−λI)v=0(A-\lambda I)v=0

For this equation to have a non-zero solution:

det(A−λI)=0det(A-\lambda I)=0

This equation is called the characteristic equation or characteristic polynomial.

The solutions of this equation are the eigenvalues of matrix (A).

Key Points

1. Relationship Between Eigenvalues and Determinant

The characteristic equation is:

det(A−λI)=0det(A-\lambda I)=0

Where:

  • (A) = original matrix
  • (I) = identity matrix
  • (\lambda) = eigenvalue

The determinant creates a polynomial equation.

The roots of this polynomial are the eigenvalues.

2. Identity Matrix

The identity matrix is used because:

Iv=vIv=v

For a 2×2 matrix:

I=[1001]I= \begin{bmatrix} 1&0\\ 0&1 \end{bmatrix}

Multiplying by identity does not change the vector.

Therefore:

λIv=λv\lambda Iv=\lambda v

3. Characteristic Equation for a 2×2 Matrix

Given:

A=[a11a12a21a22]A= \begin{bmatrix} a_{11}&a_{12}\\ a_{21}&a_{22} \end{bmatrix}

First calculate:

A−λIA-\lambda I A−λI=[a11a12a21a22]−−−−−−−−−−−−−λ[1001]A-\lambda I= \begin{bmatrix} a_{11}&a_{12}\\ a_{21}&a_{22} \end{bmatrix} ------------- \lambda \begin{bmatrix} 1&0\\ 0&1 \end{bmatrix}

Therefore:

A−λI=[a11−λa12a21a22−λ]A-\lambda I= \begin{bmatrix} a_{11}-\lambda&a_{12}\\ a_{21}&a_{22}-\lambda \end{bmatrix}

Now calculate the determinant:

det(A−λI)=0det(A-\lambda I)=0

Using the determinant formula:

(a11−λ)(a22−λ)−a12a21=0(a_{11}-\lambda)(a_{22}-\lambda)-a_{12}a_{21}=0

This equation is the characteristic polynomial.

Example / Code

Example 1: Finding Eigenvalues of a 2×2 Matrix

Given:

A=[2112]A= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix}

Step 1: Create (A-\lambda I)

The identity matrix is:

I=[1001]I= \begin{bmatrix} 1&0\\ 0&1 \end{bmatrix}

Therefore:

A−λI=[2112]−−−−−−−−−−−−−[λ00λ]A-\lambda I= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix} ------------- \begin{bmatrix} \lambda&0\\ 0&\lambda \end{bmatrix} A−λI=[2−λ112−λ]A-\lambda I= \begin{bmatrix} 2-\lambda&1\\ 1&2-\lambda \end{bmatrix}

Step 2: Calculate Determinant

det(A−λI)=0det(A-\lambda I)=0 ∣2−λ112−λ∣=0\begin{vmatrix} 2-\lambda&1\\ 1&2-\lambda \end{vmatrix} =0

Using:

∣A∣=ad−bc|A|=ad-bc

we get:

(2−λ)(2−λ)−(1)(1)=0(2-\lambda)(2-\lambda)-(1)(1)=0

Expand:

(2−λ)2−1=0(2-\lambda)^2-1=0 (4−4λ+λ2)−1=0(4-4\lambda+\lambda^2)-1=0

Simplify:

λ2−4λ+3=0\lambda^2-4\lambda+3=0

Factor:

(λ−3)(λ−1)=0(\lambda-3)(\lambda-1)=0

Step 3: Find Eigenvalues

Set each factor equal to zero:

λ−3=0\lambda-3=0 λ=3\lambda=3

and:

λ−1=0\lambda-1=0 λ=1\lambda=1

Therefore:

λ1=3,λ2=1\boxed{\lambda_1=3,\lambda_2=1}

Explanation

The characteristic equation converts the eigenvalue problem into a normal polynomial equation.

The process is:

  1. Start with:
Av=λvAv=\lambda v
  1. Move all terms to one side:
(A−λI)v=0(A-\lambda I)v=0
  1. Remove the vector by taking determinant:
det(A−λI)=0det(A-\lambda I)=0

  1. Solve the polynomial equation.

  2. The solutions are eigenvalues.

Example 2: Matrix With Different Values

Given:

A=[4123]A= \begin{bmatrix} 4&1\\ 2&3 \end{bmatrix}

Find eigenvalues.

Step 1: Calculate (A-\lambda I)

A−λI=[4−λ123−λ]A-\lambda I= \begin{bmatrix} 4-\lambda&1\\ 2&3-\lambda \end{bmatrix}

Step 2: Characteristic Equation

det(A−λI)=0det(A-\lambda I)=0 ∣4−λ123−λ∣=0\begin{vmatrix} 4-\lambda&1\\ 2&3-\lambda \end{vmatrix} =0

Using determinant formula:

(4−λ)(3−λ)−(1)(2)=0(4-\lambda)(3-\lambda)-(1)(2)=0

Expand:

12−4λ−3λ+λ2−2=012-4\lambda-3\lambda+\lambda^2-2=0

Simplify:

λ2−7λ+10=0\lambda^2-7\lambda+10=0

Factor:

(λ−5)(λ−2)=0(\lambda-5)(\lambda-2)=0

Therefore:

λ1=5,λ2=2\boxed{\lambda_1=5,\lambda_2=2}

Finding Eigenvalues of 3×3 Matrices

For a 3×3 matrix:

A=[a11a12a13a21a22a23a31a32a33]A= \begin{bmatrix} a_{11}&a_{12}&a_{13}\\ a_{21}&a_{22}&a_{23}\\ a_{31}&a_{32}&a_{33} \end{bmatrix}

We calculate:

det(A−λI)=0det(A-\lambda I)=0

The result is usually a cubic polynomial:

λ3+aλ2+bλ+c=0\lambda^3+a\lambda^2+b\lambda+c=0

The roots of this polynomial are the eigenvalues.

Example: Upper Triangular Matrix

Given:

A=[410020003]A= \begin{bmatrix} 4&1&0\\ 0&2&0\\ 0&0&3 \end{bmatrix}

For triangular matrices:

The eigenvalues are the diagonal elements.

Therefore:

λ1=4\lambda_1=4 λ2=2\lambda_2=2 λ3=3\lambda_3=3

No determinant expansion is needed.

Important Properties

1. Number of Eigenvalues

An (n \times n) matrix has:

nn

eigenvalues counting repeated values.

Example:

A 3×3 matrix has three eigenvalues.

2. Repeated Eigenvalues

Sometimes the same eigenvalue appears multiple times.

Example:

(λ−2)2(λ−5)=0(\lambda-2)^2(\lambda-5)=0

Eigenvalues:

λ=2,2,5\lambda=2,2,5

3. Triangular Matrix Property

For upper or lower triangular matrices:

The eigenvalues are simply the diagonal elements.

Example:

A=[521034007]A= \begin{bmatrix} 5&2&1\\ 0&3&4\\ 0&0&7 \end{bmatrix}

Eigenvalues:

λ=5,3,7\lambda=5,3,7

Common Mistakes

  • Forgetting to subtract (\lambda I) from matrix (A).
  • Using the wrong identity matrix size.
  • Forgetting to set determinant equal to zero.
  • Making mistakes while expanding the polynomial.
  • Confusing eigenvalues with eigenvectors.
  • Forgetting that eigenvalues come from solving the characteristic equation.

Short Exam Notes (Very Concise Revision)

  • Eigenvalues are found from:
det(A−λI)=0det(A-\lambda I)=0
  • (I) is the identity matrix.
  • The result is called the characteristic polynomial.
  • The roots of the polynomial are eigenvalues.
  • For a 2×2 matrix:
(a11−λ)(a22−λ)−a12a21=0(a_{11}-\lambda)(a_{22}-\lambda)-a_{12}a_{21}=0
  • Triangular matrix eigenvalues are diagonal elements.
  • After finding eigenvalues, find eigenvectors using:
(A−λI)v=0(A-\lambda I)v=0

Finding Eigenvectors with Examples

Part 4: Finding Eigenvectors

Definition

After finding the eigenvalues of a matrix, the next step is to find the corresponding eigenvectors.

The eigenvectors are found by solving:

(A−λI)v=0(A-\lambda I)v=0

Where:

  • (A) = original matrix
  • (\lambda) = eigenvalue
  • (I) = identity matrix
  • (v) = eigenvector

The equation:

(A−λI)v=0(A-\lambda I)v=0

is a homogeneous system of equations.

The solution vectors of this system are the eigenvectors.

Key Points

Steps for Finding Eigenvectors

To find eigenvectors:

Step 1: Find eigenvalues

First solve:

det(A−λI)=0det(A-\lambda I)=0

This gives the eigenvalues.

Step 2: Substitute each eigenvalue

For every eigenvalue, solve:

(A−λI)v=0(A-\lambda I)v=0

Step 3: Find the null space

The solution space of:

(A−λI)v=0(A-\lambda I)v=0

is the eigenspace.

Any non-zero vector in this space is an eigenvector.

Example 1: Finding Eigenvectors of a 2×2 Matrix

Given:

A=[2112]A= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix}

From Part 3, we already found the eigenvalues:

λ1=3\lambda_1=3

and

λ2=1\lambda_2=1

Now find the eigenvectors.

Finding Eigenvector for (\lambda=1)

Step 1: Calculate (A-\lambda I)

A−I=[2112]−−−−−−−−−−−−−[1001]A-I= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix} ------------- \begin{bmatrix} 1&0\\ 0&1 \end{bmatrix}

Therefore:

A−I=[1111]A-I= \begin{bmatrix} 1&1\\ 1&1 \end{bmatrix}

Step 2: Solve the System

(A−I)v=0(A-I)v=0

Let:

v=[xy]v= \begin{bmatrix} x\\ y \end{bmatrix}

Then:

[1111][xy]=============[00]\begin{bmatrix} 1&1\\ 1&1 \end{bmatrix} \begin{bmatrix} x\\ y \end{bmatrix} ============= \begin{bmatrix} 0\\ 0 \end{bmatrix}

This gives:

x+y=0x+y=0

Therefore:

x=−yx=-y

Let:

y=1y=1

Then:

x=−1x=-1

Therefore:

v=[−11]v= \begin{bmatrix} -1\\ 1 \end{bmatrix}

The eigenvector corresponding to:

λ=1\lambda=1

is:

v=[−11]\boxed{ v= \begin{bmatrix} -1\\ 1 \end{bmatrix} }

Finding Eigenvector for (\lambda=3)

Step 1: Calculate (A-3I)

A−3I=[2112]−−−−−−−−−−−−−[3003]A-3I= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix} ------------- \begin{bmatrix} 3&0\\ 0&3 \end{bmatrix}

Therefore:

A−3I=[−111−1]A-3I= \begin{bmatrix} -1&1\\ 1&-1 \end{bmatrix}

Step 2: Solve

[−111−1][xy]=============[00]\begin{bmatrix} -1&1\\ 1&-1 \end{bmatrix} \begin{bmatrix} x\\ y \end{bmatrix} ============= \begin{bmatrix} 0\\ 0 \end{bmatrix}

From the first equation:

−x+y=0-x+y=0

Therefore:

y=xy=x

Let:

x=1x=1

Then:

y=1y=1

The eigenvector is:

v=[11]\boxed{ v= \begin{bmatrix} 1\\ 1 \end{bmatrix} }

Final Answer

Eigenvalues:

λ1=3,λ2=1\lambda_1=3,\lambda_2=1

Corresponding eigenvectors:

For:

λ=3\lambda=3 v=[11]v= \begin{bmatrix} 1\\ 1 \end{bmatrix}

For:

λ=1\lambda=1 v=[−11]v= \begin{bmatrix} -1\\ 1 \end{bmatrix}

Explanation

The eigenvalue tells us the scaling amount.

The eigenvector tells us the direction that remains unchanged.

For:

λ=3\lambda=3

the vector:

[11]\begin{bmatrix} 1\\ 1 \end{bmatrix}

becomes three times larger.

For:

λ=1\lambda=1

the vector:

[−11]\begin{bmatrix} -1\\ 1 \end{bmatrix}

does not change length or direction.

Example 2: Finding Eigenvectors of a 2×2 Matrix

Given:

A=[4123]A= \begin{bmatrix} 4&1\\ 2&3 \end{bmatrix}

From the characteristic equation:

λ2−7λ+10=0\lambda^2-7\lambda+10=0

we get:

(λ−5)(λ−2)=0(\lambda-5)(\lambda-2)=0

Eigenvalues:

λ1=5\lambda_1=5 λ2=2\lambda_2=2

Eigenvector for (\lambda=5)

Calculate:

A−5IA-5I [4123]−−−[5005] \begin{bmatrix} 4&1\\ 2&3 \end{bmatrix} --- \begin{bmatrix} 5&0\\ 0&5 \end{bmatrix}

[−112−2] \begin{bmatrix} -1&1\\ 2&-2 \end{bmatrix}

Solve:

[−112−2][xy]=============[00] \begin{bmatrix} -1&1\\ 2&-2 \end{bmatrix} \begin{bmatrix} x\\ y \end{bmatrix} ============= \begin{bmatrix} 0\\ 0 \end{bmatrix}

Equation:

−x+y=0 -x+y=0

Therefore:

y=x y=x

Let:

x=1 x=1

Then:

y=1 y=1

Eigenvector:

v=[11] \boxed{ v= \begin{bmatrix} 1\\ 1 \end{bmatrix} }

Eigenvector for (\lambda=2)

Calculate:

A−2I A-2I

[4123]−−−[2002] \begin{bmatrix} 4&1\\ 2&3 \end{bmatrix} --- \begin{bmatrix} 2&0\\ 0&2 \end{bmatrix}

[2121] \begin{bmatrix} 2&1\\ 2&1 \end{bmatrix}

Solve:

2x+y=0 2x+y=0

Therefore:

y=−2x y=-2x

Let:

x=1 x=1

Then:

y=−2 y=-2

Eigenvector:

v=[1−2] \boxed{ v= \begin{bmatrix} 1\\ -2 \end{bmatrix} }

Final Answer

Eigenvalues:

λ=5,2 \lambda=5,2

Eigenvectors:

For:

λ=5 \lambda=5 v=[11] v= \begin{bmatrix} 1\\ 1 \end{bmatrix}

For:

λ=2 \lambda=2 v=[1−2] v= \begin{bmatrix} 1\\ -2 \end{bmatrix}

Eigenvectors of 3×3 Matrices

For a 3×3 matrix:

A=[a∗11a∗12a∗13a∗21a∗22a∗23a∗31a∗32a_33] A= \begin{bmatrix} a*{11}&a*{12}&a*{13}\\ a*{21}&a*{22}&a*{23}\\ a*{31}&a*{32}&a\_{33} \end{bmatrix}

The same process is used:

  1. Find eigenvalues:
det(A−λI)=0 det(A-\lambda I)=0
  1. For each eigenvalue solve:
(A−λI)v=0 (A-\lambda I)v=0
  1. The solution vectors are eigenvectors.

Example 3: 3×3 Matrix

Given:

A=[410020003] A= \begin{bmatrix} 4&1&0\\ 0&2&0\\ 0&0&3 \end{bmatrix}

Eigenvalues are diagonal values:

λ=4,2,3 \lambda=4,2,3

For (\lambda=4)

Calculate:

A−4I=[0100−2000−1] A-4I= \begin{bmatrix} 0&1&0\\ 0&-2&0\\ 0&0&-1 \end{bmatrix}

The equations are:

y=0 y=0 z=0 z=0

The variable (x) is free.

Let:

x=1 x=1

Eigenvector:

v=[100] v= \begin{bmatrix} 1\\ 0\\ 0 \end{bmatrix}

For (\lambda=2)

Calculate:

A−2I=[210000001] A-2I= \begin{bmatrix} 2&1&0\\ 0&0&0\\ 0&0&1 \end{bmatrix}

Equations:

2x+y=0 2x+y=0 z=0 z=0

Therefore:

y=−2x y=-2x

Let:

x=1 x=1

Eigenvector:

v=[1−20] v= \begin{bmatrix} 1\\ -2\\ 0 \end{bmatrix}

Common Mistakes

  • Finding eigenvectors before eigenvalues.
  • Forgetting to solve:
(A−λI)v=0 (A-\lambda I)v=0
  • Choosing the zero vector as an eigenvector.
  • Forgetting that multiple eigenvectors can exist for one eigenvalue.
  • Making calculation errors during row reduction.
  • Not simplifying eigenvectors.

Short Exam Notes (Very Concise Revision)

  • Eigenvectors are found after eigenvalues.
  • Use:
(A−λI)v=0 (A-\lambda I)v=0
  • Eigenvectors are the null space of:
A−λI A-\lambda I

  • Steps:

    1. Find eigenvalues.
    2. Substitute each eigenvalue.
    3. Solve the homogeneous system.
    4. Obtain non-zero solution vectors.

  • Eigenvectors are not unique; any scalar multiple is also an eigenvector.

Example:

If:

v=[12] v= \begin{bmatrix} 1\\ 2 \end{bmatrix}

then:

2v=[24] 2v= \begin{bmatrix} 2\\ 4 \end{bmatrix}

is also an eigenvector.

Special Cases, Practice Questions, and Exam Revision Notes

Part 5: Special Cases + Practice Questions + Exam Revision Notes

Definition

Eigenvalues and eigenvectors have several special cases that help simplify calculations and understand matrix behavior.

Some matrices have special structures where eigenvalues or eigenvectors can be found more easily.

Important special cases include:

  • Upper triangular matrices
  • Lower triangular matrices
  • Identity matrices
  • Repeated eigenvalues
  • Zero eigenvalues
  • Symmetric matrices

Key Points

1. Eigenvalues of Triangular Matrices

Definition

A triangular matrix is a matrix where all elements below or above the main diagonal are zero.

There are two types:

Upper triangular matrix

Example:

A=[421035007] A= \begin{bmatrix} 4&2&1\\ 0&3&5\\ 0&0&7 \end{bmatrix}

Lower triangular matrix

Example:

A=[500230146] A= \begin{bmatrix} 5&0&0\\ 2&3&0\\ 1&4&6 \end{bmatrix}

Important Rule

For any triangular matrix:

Eigenvalues=Diagonal elements \text{Eigenvalues}=\text{Diagonal elements}

Example:

A=[421035007] A= \begin{bmatrix} 4&2&1\\ 0&3&5\\ 0&0&7 \end{bmatrix}

The eigenvalues are:

λ1=4 \lambda_1=4 λ2=3 \lambda_2=3 λ3=7 \lambda_3=7

No determinant expansion is required.

Explanation

The characteristic equation of a triangular matrix becomes:

det(A−λI)=0 det(A-\lambda I)=0

Because the matrix is triangular:

det(A−λI) det(A-\lambda I)

is simply the product of diagonal elements:

(a∗11−λ)(a∗22−λ)(a_33−λ)=0 (a*{11}-\lambda)(a*{22}-\lambda)(a\_{33}-\lambda)=0

Therefore, the eigenvalues are the diagonal values.

2. Identity Matrix Eigenvalues

Definition

The identity matrix is:

I=[1001] I= \begin{bmatrix} 1&0\\ 0&1 \end{bmatrix}

For any vector:

Iv=v Iv=v

Comparing with:

Av=λv Av=\lambda v

We get:

λ=1 \lambda=1

Example

For:

I=[100010001] I= \begin{bmatrix} 1&0&0\\ 0&1&0\\ 0&0&1 \end{bmatrix}

All eigenvalues are:

λ=1,1,1 \lambda=1,1,1

Every non-zero vector is an eigenvector.

3. Zero Eigenvalue

Definition

If:

λ=0 \lambda=0

then:

Av=0 Av=0

This means the matrix transforms an eigenvector into the zero vector.

Important Property

A matrix has a zero eigenvalue if:

det(A)=0 det(A)=0

This means:

  • Matrix is singular.
  • Inverse does not exist.
  • Columns are linearly dependent.

Example

Given:

A=[1224] A= \begin{bmatrix} 1&2\\ 2&4 \end{bmatrix}

Calculate determinant:

det(A)=(1)(4)−(2)(2) det(A)=(1)(4)-(2)(2) =4−4 =4-4 =0 =0

Therefore:

λ=0 \lambda=0

is one of the eigenvalues.

4. Repeated Eigenvalues

Definition

Sometimes the same eigenvalue appears more than once in the characteristic polynomial.

Example:

(λ−2)2(λ−5)=0 (\lambda-2)^2(\lambda-5)=0

Eigenvalues:

λ1=2 \lambda_1=2 λ2=2 \lambda_2=2 λ3=5 \lambda_3=5

Important Note

A repeated eigenvalue does not always have multiple independent eigenvectors.

There are two possibilities:

Case 1: Enough eigenvectors exist

The matrix can be diagonalized.

Case 2: Not enough eigenvectors exist

The matrix cannot be diagonalized.

5. Symmetric Matrix Property

Definition

A matrix is symmetric if:

A=AT A=A^T

Example:

A=[2113] A= \begin{bmatrix} 2&1\\ 1&3 \end{bmatrix}

The matrix is symmetric because:

AT=A A^T=A

Important Properties of Symmetric Matrices

For symmetric matrices:

  • All eigenvalues are real.
  • Eigenvectors are orthogonal.
  • They can be diagonalized easily.

These properties are very important in machine learning.

Practice Questions

Question 1

Find the eigenvalues of:

A=[300−110533] A= \begin{bmatrix} 3&0&0\\ -1&1&0\\ 5&3&3 \end{bmatrix}

Solution

The matrix is lower triangular.

Eigenvalues are the diagonal elements:

λ1=3 \lambda_1=3 λ2=1 \lambda_2=1 λ3=3 \lambda_3=3

Final answer:

λ=3,1,3 \boxed{\lambda=3,1,3}

Question 2

Find eigenvalues of:

A=[314026005] A= \begin{bmatrix} 3&1&4\\ 0&2&6\\ 0&0&5 \end{bmatrix}

Solution

The matrix is upper triangular.

Therefore:

λ1=3 \lambda_1=3 λ2=2 \lambda_2=2 λ3=5 \lambda_3=5

Final answer:

λ=3,2,5 \boxed{\lambda=3,2,5}

Question 3

Find eigenvalues and eigenvectors of:

A=[200−110533] A= \begin{bmatrix} 2&0&0\\ -1&1&0\\ 5&3&3 \end{bmatrix}

Step 1: Find Eigenvalues

The matrix is lower triangular.

Therefore:

λ=2,1,3 \lambda=2,1,3

Step 2: Find Eigenvectors

For each eigenvalue solve:

(A−λI)v=0 (A-\lambda I)v=0

The process:

For:

λ=1 \lambda=1

solve:

(A−I)v=0 (A-I)v=0

For:

λ=2 \lambda=2

solve:

(A−2I)v=0 (A-2I)v=0

For:

λ=3 \lambda=3

solve:

(A−3I)v=0 (A-3I)v=0

Question 4

Find eigenvalues of:

A=[8−62−67−42−43] A= \begin{bmatrix} 8&-6&2\\ -6&7&-4\\ 2&-4&3 \end{bmatrix}

Given:

λ=0,3,15 \lambda=0,3,15

Find corresponding eigenvectors using:

(A−λI)v=0 (A-\lambda I)v=0

Question 5

Find eigenvalues and eigenvectors:

A=[6−22−23−12−13] A= \begin{bmatrix} 6&-2&2\\ -2&3&-1\\ 2&-1&3 \end{bmatrix}

Given eigenvalues:

λ=2,2,8 \lambda=2,2,8

Find eigenvectors for each eigenvalue.

Complete Eigenvalue and Eigenvector Algorithm

Step 1: Check Matrix Type

Ask:

  • Is it triangular?
  • Is it symmetric?
  • Is it identity?

Special structures may simplify the solution.

Step 2: Find Eigenvalues

Use:

det(A−λI)=0 det(A-\lambda I)=0

Solve the characteristic polynomial.

Step 3: Find Eigenvectors

For every eigenvalue:

(A−λI)v=0 (A-\lambda I)v=0

Find the null space.

Step 4: Verify

Check:

Av=λv Av=\lambda v

If both sides are equal, the eigenvector is correct.

Common Mistakes

  • Forgetting that eigenvalues come from:
det(A−λI)=0 det(A-\lambda I)=0
  • Using the wrong identity matrix size.
  • Finding eigenvectors before eigenvalues.
  • Accepting the zero vector as an eigenvector.
  • Forgetting repeated eigenvalues.
  • Assuming repeated eigenvalues always have multiple eigenvectors.
  • Forgetting triangular matrices use diagonal elements.
  • Not checking the final answer using:
Av=λv Av=\lambda v

Short Exam Notes (Very Concise Revision)

Norms

  • L1 norm:
∣∣x∣∣_1=∑∣xi∣ ||x||\_1=\sum |x_i|
  • L2 norm:
∣∣x∣∣_2=∑xi2 ||x||\_2=\sqrt{\sum x_i^2}
  • Frobenius norm:
∣∣A∣∣F=∑i∑j∣aij∣2 ||A||_F=\sqrt{\sum_i\sum_j |a_{ij}|^2}

Determinant

  • Only defined for square matrices.
  • If:
det(A)=0 det(A)=0

matrix is singular.

  • If:
det(A)≠0 det(A)\neq0

matrix is invertible.

Eigenvalues

Found using:

det(A−λI)=0 det(A-\lambda I)=0

Eigenvectors

Found using:

(A−λI)v=0 (A-\lambda I)v=0

Important Properties

  • Triangular matrix → eigenvalues are diagonal elements.
  • Identity matrix → eigenvalue is 1.
  • Zero determinant → zero eigenvalue exists.
  • Symmetric matrix → real eigenvalues and orthogonal eigenvectors.
  • Eigenvectors are not unique; any scalar multiple is also valid.

Final Exam Revision Summary

The complete process:

  1. Given matrix (A).
  2. Find:
det(A−λI)=0 det(A-\lambda I)=0
  1. Solve for eigenvalues.
  2. For each eigenvalue solve:
(A−λI)v=0 (A-\lambda I)v=0
  1. Obtain eigenvectors.
  2. Verify:
Av=λv Av=\lambda v

Eigenvalues tell how much a direction changes.

Eigenvectors tell which directions remain unchanged.

Together, eigenvalues and eigenvectors describe the fundamental behavior of a matrix transformation.