The short answer
What does entropy measure, and how can a probability model reduce the average length of a message? Work through four invented symbols, construct a decodable code and keep the source’s uncertainty separate from the meaning of its messages.
Original comparison
Three quantities that should not share a label
A representation is evaluated against a source and a channel, with units kept explicit.
| Country | What is specified | What it answers |
|---|---|---|
| Entropy | Probabilities and a logarithm base | Average uncertainty per symbol in that source model. |
| Code length | A codeword for every allowed symbol | Bits used for a particular message or on average. |
| Channel rate | A physical or idealised transmission model | How many encoded bits can be sent per unit time. |
Chronology
A timeline of the turning points
Two-part paper
Publication developed discrete and continuous communication models.
Source uncertainty
Entropy and noiseless coding connect a statistical source with a transmission requirement.
A chosen model
The four probabilities and code are original teaching material rather than historical experimental data.
Start with the source, not the importance of a sentence
Shannon’s paper begins by asking how a selected message can be reproduced at another point. Its source is described through the possibilities it can produce and their statistical structure. A short statement about an important event is not automatically assigned more entropy than a long unimportant one. For the engineering problem, the relevant question is how uncertain the selection was within the specified model. Meaning remains important to people using a system, but the mathematical quantity answers a different question. Shannon’s opening also credits Nyquist and Hartley, locating the paper within an existing communications research tradition.
State four probabilities before doing any arithmetic
Our invented source emits A, B, C and D independently with probabilities 1/2, 1/4, 1/8 and 1/8. These sum to one. The values are assumptions for this exercise, not a measured distribution of English letters. In base two, the surprise assigned to an event with probability p is −log₂(p). A therefore has one bit of surprise, B two bits, and C and D three bits each. A rare symbol receives a longer numerical description because its arrival is less predictable in this model, not because the character carries a more valuable idea.
Sources for this section: [1]
Calculate the average uncertainty
Entropy is the probability-weighted average of those values: H = −Σp log₂(p). For this source it is (1/2 × 1) + (1/4 × 2) + (1/8 × 3) + (1/8 × 3) = 1.75 bits per symbol. There are four possible symbols, so a fixed-length representation can use two bits for every one. The difference between two and 1.75 is not a contradiction. The first counts a uniform allocation of four labels; the second uses the unequal probabilities. A zero-probability term contributes zero under the usual limiting convention rather than requiring a finite codeword for an event excluded by the model.
Sources for this section: [1]
Build a prefix code and test its boundary rule
Assign A = 0, B = 10, C = 110 and D = 111. No complete codeword begins another complete codeword, so a decoder can recognise each symbol without an extra separator. Its expected length is 1.75 bits, matching the entropy in this particular dyadic example. That equality is special; arbitrary source probabilities need not give integer code lengths equal to every surprise value. Shannon’s noiseless coding discussion addresses how efficient encodings approach the relevant limit under its conditions. The code here is a newly constructed example, not a claim to reproduce an apparatus or historical message from Bell Labs.
Sources for this section: [1]
Count one block without confusing it with an average guarantee
Consider the invented block AABACABD. It contains four As, two Bs, one C and one D, so this code produces 00100110010111, fourteen bits. A two-bit fixed code requires sixteen bits. The reduction is two bits out of sixteen, or 12.5 per cent, excluding any headers, codebook transmission, framing or error protection. Different short blocks can fare differently. Eight Ds would use twenty-four bits under the prefix code and sixteen under the fixed code. Efficient on average does not mean shorter for every possible block. The expected result depends on the source assumptions and the scope of the cost being counted.
Sources for this section: [1]
Keep a source rate separate from a channel rate
Entropy above is in bits per source symbol. A channel rate is in bits per second, a different unit. In an idealised noiseless link carrying fourteen encoded bits per second, an average of 1.75 bits per symbol corresponds to eight source symbols per second, before overhead. A fixed two-bit code would carry seven. That arithmetic assumes the stated link and source; it is not a measurement of internet speed. The later noisy-channel analysis raises a further question about reliable communication under noise. Neither a short code nor the entropy calculation alone specifies the required error-control system or guarantees exact recovery over a damaged link.
Sources for this section: [1]
Original exercise: decode, change the model and challenge a claim
Decode 00100110010111 using the stated prefix rule. Calculate the weighted length, the eight-symbol block saving and the cost of eight Ds. Now suppose all four symbols are equally probable while the same code is retained: what is the new average length, and would a fixed two-bit code be preferable? Finally assess this claim: because the code reaches entropy for the first model, it proves the meaning of every message is preserved and that fourteen bits are sufficient for every eight-symbol input. Answer from the model, code and units rather than from Shannon’s reputation.
Sources for this section: [1]
Reference answer: a code can be correct and still be the wrong choice
The decoded block is AABACABD. Under the original probabilities the average length is 1.75; this particular block uses fourteen rather than sixteen bits, saving 12.5 per cent of the fixed-code payload. Eight Ds cost twenty-four bits. With a uniform source, the retained code averages (1 + 2 + 3 + 3)/4 = 2.25 bits per symbol, while the fixed code stays at two. The code remains decodable, but it no longer matches the new source efficiently. No result here measures semantic fidelity after translation, human understanding or actual channel errors. The lesson is operational: state the source, test the representation, count the full relevant cost and revisit the design when assumptions change.
Sources for this section: [1]
Continue the chronology
Related history guides
Shannon’s 1938 Relay Algebra: Read the Symbols and Simplify a Circuit
Follow how an algebraic representation becomes a model that can be checked.
Read the timeline1945–1947The EDVAC First Draft (1945): Why Store Instructions in Memory?
Distinguish an electronic computer’s organisation from a source code or switching network.
Read the timelineConnected biographies
People behind the work
1916–2001
Claude Shannon
Mathematician, electrical engineer and information theoristClaude Shannon connected abstract probability and symbolic logic with engineering problems. His work gave circuit designers a calculus and communication engineers a way to quantify uncertainty, coding limits and reliable transmission.Read biographyQuick answers
Common questions
- Did Shannon invent the word bit?
- The 1948 paper credits John W. Tukey for the word. Shannon made binary information central to a quantitative communication theory.
- Does the example reproduce Shannon’s original data?
- No. It is an original exercise with specified probabilities and codewords; the paper supplies the historical mathematical framework.
Primary and institutional sources
How this history was checked
Read the linked original documents and institutional records alongside the separately identified teaching exercise. This article was reviewed on 27 October 2026.