Tag: Asymptotically Efficient Estimators

Key focus: Applying Cramér-Rao Lower Bound (CRLB) for vector parameter estimation. Know about covariance matrix, Fisher information matrix & CRLB matrix.

CRLB for Vector Parameter Estimation

CRLB for scalar parameter estimation was discussed in previous posts. The same concept is extended to vector parameter estimation.

Consider a set of deterministic parameters that we wish to estimate.
The estimate is denoted in vector form as, .

Assume that the estimate is unbiased .

Covariance Matrix

For the scalar parameter estimation, the variance of the estimate was considered. For vector parameter estimation, the covariance of the vector of estimates are considered.

The covariance matrix for the vector of estimates is given by

For example, if and C are the unknown parameters to be estimated, then the  covariance matrix for the parameter vector is given by

Fisher Information Matrix

For the scalar parameter estimation, Fisher Information was considered. Same concept is extended for the vector case and is called the Fisher Information Matrix . The ijth element of the Fisher Information Matrix (evaluated at the true values of the parameter vector) is given by

CRLB Matrix

Under the same regularity condition (as that of the scalar parameter estimation case),

the CRLB Matrix is given by the inverse of the Fisher Information Matrix

Note: For the scale parameter estimation, the CRLB was shown to be the reciprocal of the Fisher Information.

This implies that the covariance of the parameters (diagonal elements) are bound by the CRLB as

More generally, the condition given above is represented as

Note: The word positive-semi-definite is the matrix equivalent of saying that a value is greater than or equal to zero. Similarly, the term positive-definite is roughly equivalent of saying that something is definitely greater than zero or definitely positive.

Emphasize was place on diagonal elements in the Fisher Information Matrix. The effect of off-diagonal elements should also be considered.

For further reading

[1] Debrati et al,“A Novel Frequency Synchronization Algorithm and its Cramer Rao Bound in Practical UWB Environment for MB-OFDM Systems”, RADIOENGINEERING, VOL. 18, NO. 1, APRIL 2009.↗

Similar topics:

[1]	An Introduction to Estimation Theory
[2]	Bias of an Estimator
[3]	Minimum Variance Unbiased Estimators (MVUE)
[4]	Maximum Likelihood Estimation
[5]	Maximum Likelihood Decoding
[6]	Probability and Random Process
[7]	Likelihood Function and Maximum Likelihood Estimation (MLE)
[8]	Score, Fisher Information and Estimator Sensitivity
[9]	Introduction to Cramer Rao Lower Bound (CRLB)
[10]	Cramer Rao Lower Bound for Scalar Parameter Estimation
[11]	Applying Cramer Rao Lower Bound (CRLB) to find a Minimum Variance Unbiased Estimator (MVUE)
[12]	Efficient Estimators and CRLB
[13]	Cramer Rao Lower Bound for Phase Estimation
[14]	Normalized CRLB - an alternate form of CRLB and its relation to estimator sensitivity
[15]	Cramer Rao Lower Bound (CRLB) for Vector Parameter Estimation
[16]	The Mean Square Error – Why do we use it for estimation problems
[17]	How to estimate unknown parameters using Ordinary Least Squares (OLS)
[18]	Essential Preliminary Matrix Algebra for Signal Processing
[19]	Why Cholesky Decomposition ? A sample case:
[20]	Tests for Positive Definiteness of a Matrix
[21]	Solving a Triangular Matrix using Forward & Backward Substitution
[22]	Cholesky Factorization - Matlab and Python
[23]	LTI system models for random signals – AR, MA and ARMA models
[24]	Comparing AR and ARMA model - minimization of squared error
[25]	Yule Walker Estimation
[26]	AutoCorrelation (Correlogram) and persistence – Time series analysis
[27]	Linear Models - Least Squares Estimator (LSE)
[28]	Best Linear Unbiased Estimator (BLUE)

Books by the author:

Wireless Communication Systems in Matlab Second Edition(PDF) Note: There is a rating embedded within this post, please visit this post to rate it. $14.99 – Add to Cart Checkout Added to cart	Digital Modulations using Python (PDF ebook) Note: There is a rating embedded within this post, please visit this post to rate it. $14.99 – Add to Cart Checkout Added to cart	Digital Modulations using Matlab (PDF ebook) Note: There is a rating embedded within this post, please visit this post to rate it. $14.99 – Add to Cart Checkout Added to cart
Hand-picked Best books on Communication Engineering Best books on Signal Processing

 

Key focus: Derive the Cramer-Rao lower bound for phase estimation applied to DSB transmission. Find out if an efficient estimator actually exists for phase estimation.

Problem formulation

Consider the DSB carrier frequency estimation problem given in the introductory chapter to estimation theory. A message is sent across a channel modulated by a sinusoidal carrier with carrier frequency = f_c and amplitude= A. The transmitted signal gets affected by zero-mean AWGN noise when it travels across the medium. The receiver receives the signal and digitizes it for further processing.

To recover the message at the receiver, one has to know every details of the sinusoid:
1) Amplitude-A
2) Carrier Frequency – f_c
3) Any uncertainty in its phase – ϕ_c.

Given a set of digitized samples x[n] and assuming that both amplitude and carrier frequency are known, we are tasked with the objective of estimating the phase of the embedded sinusoid (cosine wave). For analyzing this scenario we should have a model to begin with.

The digitized samples at the receiver are modeled as

Here A and f_c are assumed to be known and w[n] is an AWGN noise with mean μ=0 and variance=σ². We will use CRLB and try to find an efficient estimator to estimate the phase component.

CRLB for Phase Estimation:

As a pre-requisite to this article, readers are advised to go through the previous article on “Steps to find CRLB”

In order to derive CRLB, we need to have a PDF (Probability Density Function) to begin with. Since the underlying noise is modeled as an AWGN noise with mean μ=0 and variance=σ², the PDF of the observed sample that gets affected by this noise is given by a multivariate Gaussian distribution function.

The sample mean is given by

The PDF is re-written as

Since the observed samples x[n] are fixed in the above equation, we will use the likelihood notation instead of PDF notation. That is, is simply rewritten in terms of log likelihood as . The log likelihood function is given by

For simplicity,we will denote ϕ_c as ϕ. Next, take the first partial derivative of log likelihood function with respect to ϕ.

Taking the second partial derivative of the log likelihood function,

Since the above term is still dependent on the observed samples \(x[n]\), take expectation of the entire equation to average out the variations.

Let’s derive the terms like fisher information, CRLB and find out whether we can find an efficient estimator from the equations.

Fisher Information:

The Fisher Information for the given problem is

Cramer Rao Lower Bound:

The CRLB is the reciprocal of Fisher Information.

The variance of any estimator estimating the phase of the carrier for given problem will always be higher than this CRLB. That is,

As we can see from the above result, that the variance of the estimates as . Such estimators are called Asymptotically Efficient Estimators.

An efficient estimator exists if and only if the first partial derivative of log likelihood function can be written in the form

Re-writing our earlier result,

We can clearly see that the above two equations are not having the same form. Thus, an efficient estimator does not exist for this problem.

Rate this article: Note: There is a rating embedded within this post, please visit this post to rate it.

For further reading

[1] Steven M. Kay, “Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory”, ISBN: 978-0133457117, Prentice Hall, Edition 1, 1993.↗

Related Topics

[1]	An Introduction to Estimation Theory
[2]	Bias of an Estimator
[3]	Minimum Variance Unbiased Estimators (MVUE)
[4]	Maximum Likelihood Estimation
[5]	Maximum Likelihood Decoding
[6]	Probability and Random Process
[7]	Likelihood Function and Maximum Likelihood Estimation (MLE)
[8]	Score, Fisher Information and Estimator Sensitivity
[9]	Introduction to Cramer Rao Lower Bound (CRLB)
[10]	Cramer Rao Lower Bound for Scalar Parameter Estimation
[11]	Applying Cramer Rao Lower Bound (CRLB) to find a Minimum Variance Unbiased Estimator (MVUE)
[12]	Efficient Estimators and CRLB
[13]	Cramer Rao Lower Bound for Phase Estimation
[14]	Normalized CRLB - an alternate form of CRLB and its relation to estimator sensitivity
[15]	Cramer Rao Lower Bound (CRLB) for Vector Parameter Estimation
[16]	The Mean Square Error – Why do we use it for estimation problems
[17]	How to estimate unknown parameters using Ordinary Least Squares (OLS)
[18]	Essential Preliminary Matrix Algebra for Signal Processing
[19]	Why Cholesky Decomposition ? A sample case:
[20]	Tests for Positive Definiteness of a Matrix
[21]	Solving a Triangular Matrix using Forward & Backward Substitution
[22]	Cholesky Factorization - Matlab and Python
[23]	LTI system models for random signals – AR, MA and ARMA models
[24]	Comparing AR and ARMA model - minimization of squared error
[25]	Yule Walker Estimation
[26]	AutoCorrelation (Correlogram) and persistence – Time series analysis
[27]	Linear Models - Least Squares Estimator (LSE)
[28]	Best Linear Unbiased Estimator (BLUE)

Books by the author

Wireless Communication Systems in Matlab Second Edition(PDF) Note: There is a rating embedded within this post, please visit this post to rate it. $14.99 – Add to Cart Checkout Added to cart	Digital Modulations using Python (PDF ebook) Note: There is a rating embedded within this post, please visit this post to rate it. $14.99 – Add to Cart Checkout Added to cart	Digital Modulations using Matlab (PDF ebook) Note: There is a rating embedded within this post, please visit this post to rate it. $14.99 – Add to Cart Checkout Added to cart
Hand-picked Best books on Communication Engineering Best books on Signal Processing