# Score, Fisher Information and Estimator Sensitivity

(3 votes, average: 4.33 out of 5)

As we have seen in the previous articles, that the estimation of a parameter from a set of data samples depends strongly on the underlying PDF. The accuracy of the estimation is inversely proportional to the variance of the underlying PDF. That is, less the variance of PDF more is the accuracy of estimation and vice versa. In other words, the estimation accuracy depends on the sharpness of the PDF curve. Sharper the PDF curve more is the accuracy.

In geometry, given any curve, the gradient (also called slope) of the curve is zero at maximum and minimum points of the curve. Gradient of a function (representing a curve) is calculated by its first derivative. The gradient of log likelihood function is called score and is used to find Maximum Likelihood estimate of a parameter.

Denoting the score as $$u(\theta)$$,

$$u(\theta) =\frac{\partial }{\partial \theta} \left[ ln L(\theta) \right ]$$

At the MLE point, where the true value of the parameter $$\theta$$ is equal to the ML estimate $$\hat{\theta}$$ the gradient is zero. Thus equating the score to zero and finding the corresponding $$\hat{\theta}$$ gives the ML estimate of $$\theta$$ (Provided the Log Likelihood Function is a concave curve).

$$u(\hat{\theta}) = 0$$

## Curvature and Fisher Information :

In geometry, the sharpness of a curve is measured by its Curvature. The sharpness of a PDF curve is influenced by its variance. More the variance less is the sharpness and vice versa. The accuracy of the estimator is measure by the sharpness of the underlying PDF curve. In differential geometry, the curvature is related to second derivative of a function.

The mean of the score evaluated at ML estimate (or true value of estimate) $$\theta$$ is zero. This gives,

$$E[u(\theta)] = 0$$

Under this regularity condition that the expectation of the score is zero, the variance of the score is called Fisher Information. That is the expectation of second derivative of log likelihood function is called Fisher Information. It measures the sharpness of the log likelihood function. More the value of Fisher Information; more is the sharpness of the curve and vice versa. So if we can calculate the Fisher Information of a log likelihood function, then we can know more about the accuracy or sensitivity of the estimator with respect to the parameter to be estimated.

The Fisher Information denoted by $$I(\theta)$$ is given by the variance of the score.

Here the * operator indicates the operation of taking complex conjugate. The negative sign in the above equation is introduced to bring inverse relationship between variance and the Fisher Information (i.e. Fisher Information will be high for log likelihood functions that have low variance). As we can see from the above equation, that the Fisher Information is related to the second derivative (Curvature or Sharpness) of the log likelihood function. The $$I(\theta)$$ computed above is also called Observed Fisher Information