Matrix Algebra for Signal Processing

by

Key focus : Essential matrix algebra: formation of matrices, determinants, rank, inverse & transpose of matrix and solving simultaneous equations.

I thought of making a post on Cholesky Decomposition, which is a very essential technique in digital signal processing applications like generating correlated random variables, solving linear equations, channel estimation etc.., But jumping straight to the topic of Cholesky Decomposition will leave many flummoxed/confused. So I decided to touch on some essentials in basic matrix algebra before taking up advanced topics.

Prerequisite:

Basic knowledge on topics like formation of matrices, determinants, rank of a matrix, inverse & transpose of matrix, solving a system of simultaneous equation using matrix algebra.

The prerequisites listed above being fulfilled, you will learn different types of matrices in this post.

1. Vector

A matrix with only one row or one column.

\text{Row Vector with 5 elements: } A = \begin{bmatrix}a & b & c & d & e \end{bmatrix}

\text{Column Vector with 5 elements: } B = \begin{bmatrix}a \\ b \\ c \\ d \\ e \end{bmatrix}

2. Transpose of a Matrix

Transpose of a matrix is formed by interchanging the elements from row to columns. For example, the first row of the matrix becomes first column in the transpose matrix, second row of the matrix becomes second column in the transpose matrix and so on.

A = \begin{bmatrix} a & b & c \\ d & e & f\\ g & h & i \end{bmatrix} \;\;\; A^T = \begin{bmatrix} a & d & g \\ b & e & h\\ c & f & i \end{bmatrix}

(A^T)_{ij} = A_{ji}

3. Square Matrix:

Matrix with equal number of rows and columns.
(Note: The sample matrices shown below are of 3×3 dimension. They can be readily extended to nxn dimension)

\begin{bmatrix} a & b & c \\ d & e & f\\ g & h & i \end{bmatrix}

Square Matrix is further classified into

4. Symmetric Matrix :

If a square matrix and its transpose are equal, then it is called a symmetric square matrix or simply symmetric matrix.

A = \begin{bmatrix} a & b & c \\ b & d & e\\ c & e & f \end{bmatrix} \;\;\;\;\;A^T = \begin{bmatrix} a & b & c \\ b & d & e\\ c & e & f \end{bmatrix} \;\;\;\;\;\Rightarrow A = A^T

5. Diagonal Matrix

A special kind of symmetric matrix, with zeros in off-diagonal locations.

A = \begin{bmatrix} a & 0 & 0 \\ 0 & b & 0\\ 0 & 0 & c \end{bmatrix}

6. Scalar matrix

A special kind of diagonal matrix, with the equal values at the diagonal

A = \begin{bmatrix} a & 0 & 0 \\ 0 & a & 0\\ 0 & 0 & a \end{bmatrix}

7. Identity Matrix

It is a special type of scalar matrix, where the leading diagonals are one. It is denoted by I

I = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix}

I = \begin{bmatrix} 1 & 0 & 0 & 0 & 0\\ 0 & 1 & 0 & 0 & 0\\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 1\end{bmatrix}

8. Inverse of a Matrix or the notion of non-Singular Matrix:

Inverse matrices are defined for square matrices. For non-square matrices, inverses do not exist. The product of a square matrix and its inverse yields an identity matrix.
For a square matrix A.

A* A^{-1} = I

Here A^{-1} denotes the inverse of square matrix A .

It is possible to have a square matrix but not invertible. Such non-invertible square matrices are called
Singular Matrices. Matrix that are invertible are called non-singular matrices.

9. Singular Matrix:

A non-invertible square matrix. Inverse does not exist.

10. Orthogonal Matrix

An invertible square matrix, whose transpose and inverse are equal. If for an invertible square matrix A,

A^T = A^{-1} \Rightarrow A^TA^{-1} = I

then, the matrix A^T or equivalently A{-1} is called an orthogonal matrix.

11. Complex Matrix :

A matrix with complex elements.

A = \begin{bmatrix} 1+2j & 2 & 5-3j \\ -4 & -7+6j & 8+2j\\ -5j & 8 & 0 \\ 9+4j & 3 & 1-2j \end{bmatrix}

12. Complex Square Matrix

A complex matrix with with equal number of rows and columns

A = \begin{bmatrix} 1+2j & 2 & 5-3j \\ -4 & -7+6j & 8+2j\\ -5j & 8 & 0 \end{bmatrix}

13. Conjugate Transpose or Hermitian Transpose:

Similar to the transpose of a matrix defined above. But the only difference is : put the complex conjugate form of the element when interchanging from rows to columns. For a complex square matrix A , the conjugate transpose is denoted by A^*

{A^*}_{ij} = \bar{A_{ji}}

14. Hermitian Matrix:

A complex square matrix whose conjugate transpose is equal to its own. For a complex square matrix A , the Hermitian Matix is denoted by A^ {\dagger} . For a Hermitian matrix, the element of the complex square matrix located at i-th row and j-th column will be equal to the complex conjugate of the element located at j-th row and i-th column.

a_{ij} = \bar{a}_{ji}

That is,

A = A^{\dagger}

15. Triangular Matrix:

A special type of square matrix. Further classified into : lower triangular and upper triangular matrix. For a lower triangular matrix, all the elements above the main diagonal are zeros.For an upper triangular matrix, all the elements below the main diagonal are zeros.

\text{Lower Triangular : } \begin{bmatrix} a & 0 & 0 & 0\\ f & b& 0 & 0\\ g & i & c & 0 \\ h & j & k & d \end{bmatrix}

\text{Upper Triangular : } \begin{bmatrix} a & f & g & h\\ 0 & b& i & j\\ 0 & 0 & c & k \\ 0 & 0 & 0 & d \end{bmatrix}

16. Positive-Definite Matrix:

It is defined for both real matrices and complex matrices. It is matrix equivalent to positivity of real numbers. For example, the numbers +3,+5 and +6 are definitely positive. Similarly a positive definite matrix is defined to be definitely positive if it satisfies the following condition.

For real case: A n x n symmetric square real matrix – A is said to be positive definite for all x \in \mathbb{R}^{n} if and only if,

x^T A x > 0

For complex case: A n x n symmetric square complex matrix – A is said to be positive definite for all x \in \mathbb{R}^{n} if and only if,

x^* A x > 0

17. Positive-semi-definite Matrix:

It is defined for both real matrices and complex matrices. It is matrix equivalent to semi-positivity of real numbers. For example, the numbers +3,+5 and +6 are definitely positive. But the set (0,+2,+3,+7 ) is not positive definite since it includes a zero element which is neither positive nor negative. Similarly a positive semi-definite matrix is defined to be semi-definitely positive if it satisfies the following condition.

For real case: A n x n symmetric square real matrix – A is said to be positive semi-definite for all x \in \mathbb{R}^{n} if and only if,

x^T A x \geq 0

For complex case: A n x n symmetric square complex matrix – A is said to be positive semi-definite for all x \in \mathbb{R}^{n} if and only if,

x^* A x \geq 0

Note : The notation \mathbb{R} stands for a set of real numbers. The notation \mathbb{R}^{n} denotes a set of n real numbers in n-dimensional space.

18. Negative-Definite Matrix:

It is defined for both real matrices and complex matrices. It is matrix equivalent to negativity of real numbers. For example, the numbers -3,-5 and -6 are definitely negative. Similarly a negative definite matrix is defined to be definitely negative if it satisfies the following condition.

For real case: A n x n symmetric square real matrix – A is said to be negative definite for all x \in \mathbb{R}^{n} if and only if,

x^T A x < 0

For complex case: A n x n symmetric square complex matrix – A is said to be negative definite for all x \in \mathbb{R}^{n} if and only if,

x^* A x < 0

Note : The notation \mathbb{R} stands for a set of real numbers. The notation \mathbb{R}^{n} denotes a set of n real numbers in n-dimensional space.

19. Negative-semi-definite Matrix:

It is defined for both real matrices and complex matrices. It is matrix equivalent to semi-negativity of real numbers. For example, the numbers -3,-5 and -6 are definitely negative. But the set (-2,-3,0,-7 ) is not negative definite since it includes a zero element which is neither positive nor negative. Similarly a negative semi-definite matrix is defined to be semi-definitely negative if it satisfies the following condition.

For real case: A n x n symmetric square real matrix – A is said to be negative semi-definite for all x \in \mathbb{R}^{n} if and only if,

x^T A x \leq 0

For complex case: A n x n symmetric square complex matrix – A is said to be negative semi-definite for all x \in \mathbb{R}^{n} if and only if,

x^* A x \leq 0

Note : The notation \mathbb{R} stands for a set of real numbers. The notation \mathbb{R}^{n} denotes a set of n real numbers in n-dimensional space.

20. Indefinite Matrix :

A Matrix, that is neither positive definite, positive semi-definite, negative definite nor negative semi-definite is called an indefinite Matrix.

Rate this article: PoorBelow averageAverageGoodExcellent (3 votes, average: 2.00 out of 5)

External References:

[1] Vector
[2] Transpose of a Matrix
[3] Square Matrix
[4] Symmetric Matrix
[5] Diagonal Matrix
[6] Scalar Matrix
[7] Identity Matrix
[8] Inverse Matrix
[9] Singular Matrix
[10] Orthogonal Matrix
[11] Complex Matrix
[12] Complex Square Matrix
[13] Conjugate Transpose
[14] Hermitian Matrix
[15] Triangular Matirx
[16] Positive definite, negative definite and semi-definite Matrices

Post your valuable comments !!!