This post contains interactive python code which you can execute in the browser itself.
Central Limit Theorem – What is it ?
The central limit theorem (CLT) is a fundamental concept in statistics and probability theory that explains how the sum of independent and identically distributed random variables behaves. The theorem states that as the number of these variables increases, the distribution of their sum tends to become more like a normal distribution, even if the variables themselves are not normally distributed.
The form that is actually used takes a bit more care. If \(X_1,\ldots,X_N\) are i.i.d. with mean \(\mu\) and a finite variance \(\sigma^2\), the sum \(S_N\) has mean \(N\mu\) and variance \(N\sigma^2\), both of which grow with \(N\). What settles to a fixed normal curve is the standardized sum
\[Z_N = \frac{S_N – N\mu}{\sigma\sqrt{N}} = \frac{\bar{X}_N – \mu}{\sigma/\sqrt{N}}\]which converges in distribution to \(N(0,1)\) as \(N \to \infty\). The same statement is the reason a sample mean, after it is scaled by \(\sqrt{N}\), has an approximately normal sampling distribution. The variables themselves need not be normal. Finite mean and variance are the conditions used here. The underlying distribution can be anything reasonable: binomial, Poisson, exponential, Chi-Squared etc.
Why CLT ?
CLT is an important concept in statistics because it allows us to make inferences about a population based on a sample, even if we do not know the distribution of the population. It is used in many statistical techniques, such as hypothesis testing and confidence intervals.
Applications of CLT
Central limit theorem (CLT) is applied in a vast range of applications including (but not limited to) signal processing, channel modeling, random process, population statistics, engineering research, predicting the confidence intervals, hypothesis testing, etc. One such application in signal processing is – deriving the response of a cascaded series of low pass filters by applying the CLT. In the article titled ‘the central limit theorem and low-pass filters‘ the author has illustrated how the response of a cascaded series of low pass filters approaches Gaussian shape as the number of filters in the series increase [1].
Thermal noise in a resistor is the usual communications example. It is the sum of an enormous number of independent contributions from the charge carriers, so the CLT puts a Gaussian distribution on the observed voltage, which is why an AWGN model is the default in a link simulation. Shot noise is a different mechanism: charge arrives in discrete events and the count is Poisson. The Poisson distribution itself becomes approximately Gaussian once the mean count is large, again by the CLT, but it is not the same thing as thermal noise.
Law of large numbers and CLT
The law of large numbers and the central limit theorem describe two different things about the same average.
The law of large numbers is another important theorem in probability theory, which states that as the number of independent and identically distributed (iid) random variables increases, the average of those variables converges to the expected value of the distribution. In other words, as the sample size increases, the sample mean becomes more and more representative of the true population mean.
The central limit theorem, on the other hand, describes the distribution of the sum of iid random variables, and shows that as the sample size increases, the distribution of the sum approaches a normal distribution.
Both the law of large numbers and the CLT deal with the behavior of the sum or average of iid random variables as the sample size gets larger. The law of large numbers describes the behavior of the sample mean, while the CLT describes the behavior of the sum of the variables.
The law of large numbers says the sample mean converges to \(\mu\). It needs a finite mean, and it says nothing about the shape of the fluctuations. The central limit theorem describes those fluctuations: they are approximately Gaussian and they shrink as \(1/\sqrt{N}\). With a finite variance the CLT implies the law of large numbers, but the law of large numbers also holds for some distributions whose variance is infinite, where the CLT in the form above does not apply.
Demonstration using Python
For Matlab code, please refer the following book – Wireless communication systems in Matlab – by Mathuranathan Viswanathan
Each trial draws an integer uniformly from \(\{1,\ldots,k\}\). A die is \(k = 6\). A coin is \(k = 2\), with the two faces labelled 1 and 2. The mean of one draw is \((k+1)/2\) and the variance is \((k^2-1)/12\). The script standardizes the sum of \(N\) draws and overlays the \(N(0,1)\) density. Set experiment to 'coins' for the second figure. The interactive block uses the same calculation with the Agg backend, which the in-browser widget requires. Figures 1 and 2 are those standardized histograms, for dice and for coins.
Python code
The interactive block runs the same calculation. The listing underneath is there so the script can be copied without opening the widget.
# Central limit theorem, standardized sums
# Mathuranathan Viswanathan, gaussianwaves.com
import numpy as np
import matplotlib.pyplot as plt
num_terms = np.array([1, 2, 5, 10, 50, 100])
experiment = "dice" # "dice" or "coins"
faces = {"dice": 6, "coins": 2}
n_samples = 100_000
k = faces[experiment]
mu = (1 + k) / 2.0
var = (k ** 2 - 1) / 12.0
fig, axes = plt.subplots(ncols=3, nrows=2, figsize=(9, 6), constrained_layout=True)
grid = np.linspace(-4, 4, 400)
normal_pdf = np.exp(-0.5 * grid ** 2) / np.sqrt(2 * np.pi)
for i, n in enumerate(num_terms):
draws = np.random.randint(1, k + 1, size=(n, n_samples))
z = (draws.sum(axis=0) - n * mu) / np.sqrt(n * var)
ax = axes[i // 3, i % 3]
ax.hist(z, bins=40, density=True, color="#4C72B0", alpha=0.85)
ax.plot(grid, normal_pdf, color="#C44E52", lw=1.6)
ax.set_xlim(-4, 4)
ax.set_title("N=%d %s" % (n, experiment))
plt.show()
Simulation results
Reference images
