Cramér–Rao lower bound (CRLB) — scalar parameter formula & Matlab check

In one sentence: The CRLB is the smallest variance any unbiased estimator can achieve — computed from the Fisher information of your observation model.

Matlab/Python simulation: Derive Fisher information for a mean-in-AWGN example, then compare sample variance of an MLE Monte Carlo to the CRLB curve.

Key focus: Discuss scalar parameter estimation using CRLB. Estimate DC component from observed data in the presence of AWGN noise.

Consider a set of observed data samples $latex X={X_1,X_2,X_3,\cdots, X_N} $ and $latex \theta $ is the scalar parameter that is to be estimated from the observed samples. The accuracy of the estimate depends on how well the observed data is influenced by the parameter $latex \theta $. The observed data is considered as a random data whose PDF is influenced by $latex \theta $. The PDF $latex p(x,\theta) $ describes the dependence of X on $latex \theta $.

If the of PDF $latex p(x,\theta) $ depends weakly on $latex \theta $ then the estimates will be poor.If the of PDF $latex p(x,\theta) $ on depends strongly on $latex \theta $ then the estimates will be good.

As seen in the previous section, the curvature of the likelihood function (Fisher Information) is related to the concentration of PDF. More the curvature, more is the concentration of PDF, more will be accuracy of estimates. The Fisher Information is calculated from log likelihood function as,

$latex I(\theta) = -E\left [ \frac{\partial^2 ln L(\theta)}{\partial \theta^2} \right ] $

Under the regularity condition that the score of the log likelihood function is zero,

$latex E\left [ \frac{\partial ln L(\theta) }{\partial \theta } \right ] = 0 \;\;\; \forall\theta $

The inverse of the Fisher Information gives the Cramér-Rao Lower Bound (CRLB).