More on Shannon’s limit

Shannon’s limit does not depend on BER. Shannon’s limit tell us the minimum possible Eb/No required to achieve an arbitrarily small probability of error as M->∞. (M is the number of signaling levels for the modulation technique, for BPSK M=2, QPSK M=4 , and so on..).

It gives the minimum possible Eb/No that satisfies the Shannon’s-Hartley law. In otherwords, it gives the minimum possible Eb/No to achieve maximum transmission capacity (R=C, where, R is the rate of transmission and C is the channel capacity). It will not specify what BER you will get at that limit. It also will not specify which coding technique to use to achieve the limit. As the capacity is approached, the system complexity will increase drastically. So the aim of any system design is to achieve that limit. LDPC coding “nears” the Shannon’s limit but it cannot achieve it.

Shannon’s limit depends on the type of channel that is being investigated. Here the channel is assumed to be AWGN. The method to calculate the shannon’s limit for an AWGN channel is as follows..

Given B and S/N, assume we are transmitting R bits/second and we wish to ensure that R is less than the Shannon-Hartley limit C (if R>C BER will increase drastically and the entire system will fail). Then:

$latex R \leq B \times log_{2} \left( 1+ \frac{S}{N}\right)&s=2&fg=0000A0$

Now assume that we wish to transmit average energy/bit of Eb (Joules per bit) and the AWGN noise has 2-sided power spectral density N0 /2 Watts (normalised to 1 Ohm) per Hz. It follows that the signal power S = REb and the noise power N = BN0 Watts. Therefore:

$latex \frac{R}{B} \leq log_{2} \left( 1+ \frac{RE_{b}}{BN_{0}}\right)&s=2&fg=0000A0$

$latex \frac{R}{B}$ is called the bandwidth efficiency in units of bit/second/Hz.

rearranging ….

$latex 2^{\frac{R}{B}}\leq 1+ \frac{RE_{b}}{BN_{0}}&s=2&fg=0000A0$

substituting $latex \frac{Eb}{N_{0}}$ with $latex \left( \frac{Eb}{N_{0}} \right)_{min}$ and rearranging,

$latex \left( \frac{E_{b}}{N_{0}} \right)_{min} \geq \frac{ \left( 2^{\frac{R}{B}} – 1 \right)}{\left( \frac{R}{B} \right)}&s=3&fg=0000A0$

This gives the minimum possible normalised energy satisfying the Shannon-Hartley law. If we draw a graph of $latex \left( \frac{Eb}{N_{0}} \right)_{min}$ against bandwidth efficiency (R/B) we observe that $latex \left( \frac{Eb}{N_{0}} \right)_{min}$ never goes less than about 0.69 which is about -1.6 dB in logarthmic scale.Therefore if our normalised energy per bit is less than -1.6dB, we can never satisfy the Shannon-Hartley law and the designed communication system will fail completely.

See also:

[1] Shannon’s Limit and Channel Capacity
[2] Performance of Channel Codes and coding metrics
[3] Introduction to Channel Coding

Recommended Books

More Recommended Books at our Integrated Book Store

Channel Capacity & Shannon’s theorem – demystified

1 Star2 Stars3 Stars4 Stars5 Stars (9 votes, average: 4.67 out of 5)

This post is a part of the ebook : Simulation of digital communication systems using Matlab – available in both PDF and EPUB format.

A chapter dedicated to Shannon’s Theorem in the ebook
, focuses on the concept of  Channel capacity. The concept of Channel capacity is discussed first followed by an in-depth treatment of Shannon’s capacity for various channels

Shannon’s Theorem

How much data will a channel/medium carry in one second or what is the data rate supported by the channel? Let’s discover the answer for this question in detail.

Any discussion about the design of a communication system will be incomplete without mentioning Shannon’s Theorem. Shannon’s information theory tells us the amount of information a channel can carry. In other words it specifies the capacity of the channel. The theorem can be stated in simple terms as follows

  • A given communication system has a maximum rate of information C known as the channel capacity
  • If the transmission information rate R is less than C, then the data transmission in the presence of noise can be made to happen with arbitrarily small error probabilities by using intelligent coding techniques
  • To get lower error probabilities, the encoder has to work on longer blocks of signal data. This entails longer delays and higher computational requirements

The Shannon-Hartley theorem indicates that with sufficiently advanced coding techniques, transmission that nears the maximum channel capacity – is possible with arbitrarily small errors. One can intuitively reason that, for a given communication system, as the information rate increases, the number of errors per second will also increase.

Shannon – Hartley Equation

Shannon-Hartley equation relates the maximum capacity (transmission bit rate) that can be achieved over a given channel with certain noise characteristics and bandwidth. For an AWGN the maximum capacity is given by (Check Appendix A1 and A2 for derivation of Shannon-Hartley equation for AWGN cannel)

C = B \; log_2 \left( 1 + \frac{S}{N}\right)  \;\;\;\;\;\;\;\;\;\; \rightarrow (1)

Here \(C\) is the maximum capacity of the channel in bits/second otherwise called Shannon’s capacity limit for the given channel, \(B\) is the bandwidth of the channel in Hertz, \(S\) is the signal power in Watts and \(N\) is the noise power, also in Watts. The ratio \(S/N\) is called Signal to Noise Ratio (SNR). It can be ascertained that the maximum rate at which we can transmit the information without any error, is limited by the bandwidth, the signal level, and the noise level. It tells how many bits can be transmitted per second without errors over a channel of bandwidth \(B \; Hz\), when the signal power is limited to \(S \; Watts\) and is exposed to Gaussian White (uncorrelated) Noise (\(N \; Watts\)) of additive nature.

Shannon’s capacity limit is defined for the given channel. It is the fundamental maximum transmission capacity that can be achieved on a channel given any combination of any coding scheme, transmission or decoding scheme. It is the best performance limit that we hope to achieve for that channel.

The above expression for the channel capacity makes intuitive sense:

  • Bandwidth limits how fast the information symbols can be sent over the given channel
  • The SNR ratio limits how much information we can squeeze in each transmitted symbols. Increasing SNR makes the transmitted symbols more robust against noise. SNR is a function of signal quality, signal power and the characteristics of the channel. It is measured at the receiver’s front end
  • To increase the information rate, the signal-to-noise ratio and the allocated bandwidth have to be traded against each other
  • For no noise, the signal to noise ratio becomes infinite and so an infinite information rate is possible at a very small bandwidth

Thus we may trade off bandwidth for SNR. However, as the bandwidth \(B\) tends to infinity, the channel capacity does not become infinite – since with an increase in bandwidth, the noise power also increases.

The Shannon’s equation relies on two important concepts:

  • That, in principle, a trade-off between SNR and bandwidth is possible
  • That, the information capacity depends on both SNR and bandwidth

It is worth to mention two important works by eminent scientists prior to Shannon’s paper [1] , which is as follows.

Edward Amstrong’s earlier work on Frequency Modulation (FM) is an excellent proof for showing that SNR and bandwidth can be traded off against each other. He demonstrated in 1936, that it was possible to increase the SNR of a communication system by using FM at the expense of allocating more bandwidth [2]

In 1903, W.M Miner in his patent (U. S. Patent 745,734 [3]), introduced the concept of increasing the capacity of transmission lines by using sampling and time division multiplexing techniques. In 1937, A.H Reeves in his French patent (French Patent 852,183, U.S Patent 2,272,070 [4]) extended the system by incorporating a quantizer, there by paving the way for the well-known technique of Pulse Coded Modulation (PCM). He realized that he would require more bandwidth than the traditional transmission methods and used additional repeaters at suitable intervals to combat the transmission noise. With the goal of minimizing the quantization noise, he used a quantizer with a large number of quantization levels. Reeves patent relies on two important facts:

  • One can represent an analog signal (like speech) with arbitrary accuracy, by using sufficient frequency sampling, and quantizing each sample in to one of the sufficiently large pre-determined amplitude levels
  • If the SNR is sufficiently large, then the quantized samples can be transmitted with arbitrarily small errors

It is implicit from Reeve’s patent – that an infinite amount of information can be transmitted on a noise free channel of arbitrarily small bandwidth. This links the information rate with SNR and bandwidth.

Please refer [1] and [5]  for the actual proof by Shannon. A much simpler version of proof (I would rather call it an illustration) can be found at [6] } (under the section “External Links” at the end of this ebook)

Unconstrained Shannon Limit for AWGN channel

Some general characteristics of the Gaussian channel can be demonstrated. Consider that we are sending binary digits across an AWGN channel at a transmission rate \(R\) equal to the channel capacity \(C\) : \( R = C \). If the average signal power is \(S\), then the average energy per bit is \(E_b = S/C \; (Joules per bit)\), since the bit duration is \(1/C\) seconds. If the one sided noise power spectral density is \(N_0/2 \; Watts/Hertz \) (power normalized to \(1 \; \Omega \) resistance), then the total noise power is \(N_0B \; Watts \). The Shannon-Hartley equation becomes

\frac{C}{B} = log_2 \left( 1 + \frac{E_b}{N_0} \frac{C}{B}\right) \;\;\;\;\;\;\;\;\;\; \rightarrow (2)

Rearranging the equation,

\frac{E_b}{N_0} = \frac{B}{C} \left( 2^{\frac{C}{B}} – 1 \right) \;\;\;\;\;\;\;\;\;\; \rightarrow (3)

Letting \(C/B = \eta \) (the spectral efficiency in \(bits/seconds/Hz\)),

\frac{E_b}{N_0} = \frac{2^ \eta – 1}{\eta} \;\;\;\;\;\;\;\;\;\; \rightarrow (4)

A snippet of Matlab code used to plot the above relationship is given below.

Check this ebook for the matlab code – Simulation of Digital Communication systems using Matlab

The plot in the following Figure, the red dashed line in the graph represents the asymptote of \(E_b/N_0\) as the bandwidth \(B\) approaches infinity. The asymptote is at \(E_b/N_0 = ln(2) = -1.59 \; dB\). This value is called Shannon’s Limit or specifically Shannon’s power efficiency limit.


Let’s derive the Shannon’s power efficiency limit and check if it is equal to \(-1.59 \;dB\). The asymptotic value (say \(x\)), that we are seeking, is the value of \(E_b/N_0\) as the spectral efficiency \(\eta\) approaches \(0\).

x=\lim_{n \to 0}\left ( \frac{E_b}{N_0} \right )=\lim_{n \to 0}\left ( \frac{2^\eta -1 }{\eta}\right ) \;\;\;\;\;\;\;\;\;\; \rightarrow (5)

Let \(f(\eta)=2^\eta-1\) and \(g(\eta)= \eta\). As \(f(0)=g(0)=0\) and the argument of the limit becomes indeterminate (\(0/0\)), L’Hospital’s rule can be applied in this case. According to L’Hospital’s rule, if \( \lim_{\eta \to k} f(\eta)\) and \( \lim_{\eta \to k}g(\eta)\)are both zero or are both \(\pm \infty\), then for any value of \(k\)

\lim_{\eta \to k} \left( \frac{f(\eta)}{g(\eta)} \right) = \lim_{\eta \to k} \left( \frac{f'(\eta)}{g'(\eta)} \right) \;\;\;\;\;\;\;\;\;\; \rightarrow (6)

Thus, the next step boils down to finding the first derivative of \(f(\eta)\) and \(g(\eta)\). Expressing \(2^n\) in natural logarithm

2=e^{ln2} \nonumber \\
2^\eta=(e^{ln2} )^{\eta}=(e^{\eta ln2} ) \;\;\;\;\;\;\;\;\;\; \rightarrow (7)

Let \(u= \eta ln(2)\) and \(y=e^u\), then by chain rule of differentiation,

f'(\eta) =\frac{d2^\eta}{d \eta} = \frac{dy}{d \eta} = \frac{dy}{du} \frac{du}{d \eta} = e^u ln(2) = e^{\eta ln 2} ln(2) \;\;\;\;\; \rightarrow (8)

Since \(g(\eta)=\eta\), the first derivative of \(g(\eta)\) is
g'(\eta) = 1 \;\;\;\;\;\;\;\;\;\; \rightarrow (9)

Using equations \((8)\) and \((9)\) and applying L’Hospital’s rule, the Shannon’s limit is given by

x &=\lim_{\eta \to 0} \left ( \frac{f(\eta)}{g(\eta)} \right ) = \lim_{\eta \to 0} \left ( \frac{f'(\eta)}{g'(\eta)} \right ) \nonumber \\
& = \lim_{\eta \to 0} \left ( ln2\;e^{\eta ln 2} \right ) = ln(2)=0.6931 = -1.59 \; dB   \;\;\;\;\;\;\;\;\;\; \rightarrow (10)

Shannon’s power efficiency limit does not depend on BER. Shannon’s limit tells us the minimum possible \(E_b/N_0\) required for achieving an arbitrarily small probability of error as \(M \to \infty\). (\(M\) is the number of signaling levels for the modulation technique, for BPSK \(M=2\), QPSK \(M=4\) and so on…).

It gives the minimum possible \(E_b/N_0\) that satisfies the Shannon’s-Hartley law. In other words, it gives the minimum possible \(E_b/N_0\) required to achieve maximum transmission capacity (\(R=C\), where, \(R\) is the rate of transmission and \(C\) is the channel capacity). It will not specify at what BER you will get at that limit. It also will not specify which coding technique to use to achieve the limit. As the capacity is approached, the system complexity will increase drastically. So the aim of any system design is to achieve that limit. As an example, a class of codes called Low Density Parity Codes (LDPC) near the Shannon’s limit but it cannot achieve it.

The Shannon limit derived above is called absolute Shannon power efficiency Limit. It is the limit of a band-limited system irrespective of modulation or coding scheme. This is also called unconstrained Shannon power efficiency Limit. If we select a particular modulation scheme or an encoding scheme, we can calculate the constrained Shannon limit for that scheme. We will see the generic form of Shannon Equation that applies to any channel and then later develop it to find the constrained capacities for AWGN channel.

In the next sections, generic forms of unconstrained Shannon equations are discussed for different types of generic channel models. These generic equations can be utilized to find the unconstrained capacity of a particular channel type – AWGN for example (check Appendix A1, A2 and A3 for details on how to extend the generic equation to a particular channel type). Derivation of Ergodic capacity of a fading channel is available here.

1 Star2 Stars3 Stars4 Stars5 Stars (9 votes, average: 4.67 out of 5)

References :

[1] C. E. Shannon, “A Mathematical Theory of Communication”, Bell Syst. Techn. J., Vol. 27, pp.379-423, 623-656, July, October, 1948
[2]E. H. Armstrong:, “A Method of Reducing Disturbances in Radio Signaling by a System of Frequency-Modulation”, Proc. IRE, 24, pp. 689-740, May, 1936
[3] Willard M Miner, “Multiplex telephony”, US Patent, 745734, December 1903, USPTO link
[4] A.H Reeves, “Electric Signaling System”, US Patent 2272070, Feb 1942, USPTO link
[5]Shannon, C.E., “Communications in the Presence of Noise”, Proc. IRE, Volume 37 no1, January 1949, pp10-21
[6] The Scotts Guide to Electronics, “Information and Measurement”, University of Andrews – School of Physics and Astronomy – click here

 See also :

[1]   Correlative Coding – Modified Duobinary Signaling
[2]   Correlative Coding – Duobinary signaling
[3]   Derivation of expression for a Gaussian Filter with 3 dB bandwidth
[4]   Nyquist and Shannon Theorem
[5]   Correlative coding – Duobinary Signaling
[6]   Square Root Raised Cosine Filter (Matched/split filter implementation)
[7]   Introduction to Inter Symbol Interference
[8]   Derivation of Error Rate Performance of an optimum BPSK receiver in AWGN channel
[9]   Eb/N0 Vs BER for BPSK over Rician Fading Channel
[10] BER Vs Eb/N0 for QPSK modulation over AWGN
[11] BER Vs Eb/N0 for 8-PSK modulation over AWGN
[12] Simulation of M-PSK modulation techniques in AWGN channel
[13] Performance comparison of Digital Modulation techniques

Recommended Books

More Recommended Books at our Integrated Book Store