Back to the on-screen lesson · Print

The binary entropy function

The entropy $h(p)$ of a biased coin: its values, its symmetry about one half, its peak of one bit and its steep ends.

Paper packet. Every task here also exists on screen, where it is checked automatically; answers written on paper are not assessed by Nydus. When you are back at a device, enter your answers there.

1. What you will learn

By the end of this lesson you will be able to write the binary entropy function $h(p)$, read its values from a short table using the symmetry $h(p) = h(1 - p)$, order coins by their entropy from the distance of their bias to one half, state its range and where the ends of that range are attained, and use $h$ as the floor under the storage of a biased bit stream.

2. What you already have

Entropy is the average surprise of a distribution, and for two outcomes there are exactly two terms. This lesson studies that two-term sum as a function of the bias of the coin, because it returns in every channel formula later in the course.

3. Bias, symmetry, concavity

The bias of a coin is $p = P(\text{heads})$. The binary entropy function $h(p)$ is the entropy of that coin. It is symmetric about $\tfrac{1}{2}$, meaning $h(p) = h(1 - p)$, and concave, meaning every chord lies below the curve; its peak is one bit at the fair coin.

4. The binary entropy function

A coin that shows heads with probability $p$ has entropy $$h(p) = -p \log_2 p - (1 - p) \log_2 (1 - p),$$ the binary entropy function. It is $0$ at $p = 0$ and $p = 1$ (no uncertainty), rises to its single peak $h(\tfrac{1}{2}) = 1$ bit, and is symmetric, $h(p) = h(1 - p)$, because renaming heads and tails changes nothing. Some values worth remembering: $h(0.1) \approx 0.469$, $h(0.11) \approx 0.5$, $h(0.2) \approx 0.722$, $h(0.25) \approx 0.811$. The curve is concave, which is why mixing two coins never lowers the average entropy, and it is steep near the ends: a small chance of a rare event costs a surprising amount of uncertainty. $h$ reappears in the capacity of the binary symmetric channel, $1 - h(\varepsilon)$, and in counting: there are about $2^{n h(k/n)}$ binary strings of length $n$ with $k$ ones.

5. A table of $h(p)$

The two terms of $h(p) = -p \log_2 p - (1 - p) \log_2 (1 - p)$ and their sum, for the values the practice uses. Because of the symmetry $h(p) = h(1 - p)$ the table only needs $p \le \tfrac{1}{2}$: read $h(0.8)$ from the row $p = 0.2$.

Reading the graph: the arch is flat near the top, so $h(0.4) = 0.971$ and $h(0.45) = 0.993$ are both close to $1$, and steep near the ends, where $h(0.05) = 0.286$ is already more than a quarter of a bit. The dashed line is the axis of symmetry.

6. Why the arch has that shape

Differentiate: $h'(p) = \log_2 \dfrac{1 - p}{p}$, positive for $p < \tfrac{1}{2}$, zero at $p = \tfrac{1}{2}$ and negative after, so there is a single peak of height $h(\tfrac{1}{2}) = 1$. As $p \to 0$ the slope $\log_2 ((1 - p)/p)$ grows without bound: the tangent is vertical, which is why a rare event adds uncertainty so quickly.

The second derivative $h''(p) = -\dfrac{1}{p (1 - p) \ln 2}$ is negative everywhere, so $h$ is strictly concave: for any two biases $p_1, p_2$ and any weight $\lambda \in (0, 1)$, $$h(\lambda p_1 + (1 - \lambda) p_2) \ge \lambda h(p_1) + (1 - \lambda) h(p_2).$$ Mixing two coins never lowers the average entropy, and this is the one-dimensional case of the concavity of entropy that Jensen's inequality will give in general.

7. Solving the practice problems

1. Compute $h(p)$. Read both terms from the table, or compute them with $\log_2 x = \ln x / 0.6931$, then add. Keep three decimals in each term.
2. Given $h(p)$, find $h(1 - p)$. It is the same number; no computation is needed, only the symmetry.
3. Compare with $h(0.5) = 1$. One bit is the maximum and is reached only at $p = 0.5$, so any other bias gives strictly less, and the farther from $0.5$ the smaller.

Common mistakes

• Dropping the second term: at $p = 0.1$ that gives $0.332$ instead of $0.469$.
• Rounding each term to two decimals before adding; round only the sum.
• Assuming $h(0.9) < h(0.1)$ because $0.9$ is the bigger number; the two coins are equally uncertain.

8. Where this usually goes wrong

The first error is dropping the tails term: at $p = 0.1$ that gives $0.332$ instead of $0.469$, because both faces contribute. The second is believing a biased coin carries more information than a fair one; it carries less, since it is easier to predict, and the fair coin is the unique maximum at one bit. The third is assuming $h(0.9) < h(0.1)$ because $0.9$ is the bigger number; the two coins are the same coin with the faces renamed. And rounding each term to two decimals before adding shifts the third decimal; round only the sum.

9. $h(0.9)$

1. $-0.9 \log_2 0.9 \approx 0.9 \cdot 0.152 = 0.137$ and $-0.1 \log_2 0.1 \approx 0.1 \cdot 3.322 = 0.332$.

   Two terms.

2. $h(0.9) \approx 0.469$ bits, the same as $h(0.1)$ by symmetry.

   The outcome is mostly predictable, so well under $1$ bit.

10. Where $h$ is steep

1. $h(0.01) \approx 0.081$ and $h(0.05) \approx 0.286$: raising a rare event from $1\%$ to $5\%$ adds $0.2$ bits.

   Compare with the same change near the middle.

2. $h(0.45) \approx 0.993$ and $h(0.5) = 1$: the same $5\%$ near the middle adds only $0.007$ bits.

   The derivative $\log_2 \frac{1 - p}{p}$ is huge near $0$ and zero at $\tfrac{1}{2}$.

11. $h(0.25)$ by hand

1. First term: $-0.25 \log_2 0.25 = 0.25 \cdot 2 = 0.5$.

   $0.25 = 2^{-2}$, so its surprise is exactly $2$ bits.

2. Second term: $-0.75 \log_2 0.75 = 0.75 \cdot (2 - \log_2 3) = 0.75 \cdot 0.415 = 0.311$.

   $\log_2 (4/3) = \log_2 4 - \log_2 3$.

3. $h(0.25) = 0.5 + 0.311 = 0.811$ bits, and by symmetry $h(0.75) = 0.811$ too.

12. $h(0.3)$ with natural logarithms

1. $\ln 0.3 = -1.204$ and $\ln 0.7 = -0.357$.

   Any calculator gives natural logarithms; convert at the end.

2. In nats: $-(0.3 \cdot (-1.204) + 0.7 \cdot (-0.357)) = 0.361 + 0.250 = 0.611$.

3. Divide by $\ln 2 = 0.693$: $h(0.3) = 0.611 / 0.693 = 0.881$ bits, matching the table.

13. Your turn: $h(0.75)$

1. By symmetry $h(0.75) = h(0.25) = 0.25 \cdot 2 + 0.75 \log_2 (4/3)$.

2. Your turn: work this step out. Its working is at the end of the packet.

   $0.5 + 0.75 \cdot 0.415 = 0.811$ bits.

14. Guided practice

Given $h(0.1) = 0.469$ bits, fill in the three other values of the binary entropy function.

15. Guided practice

Using $h(0.05) = 0.286$, $h(0.1) = 0.469$ and $h(0.25) = 0.811$ bits, match each coin to its entropy.

16. Practice

Put these coins in order of increasing entropy, least uncertain first.

Number the steps in order (write the number in the box):

• $P(\text{heads}) = 0.01$
• $P(\text{heads}) = 0.9$
• $P(\text{heads}) = 0.7$
• $P(\text{heads}) = 0.5$

17. Practice

Which is larger, $h(0.2)$ or $h(0.5)$, and why?

• $h(0.5) = 1$ is larger: the fair coin is the hardest to predict
• $h(0.2)$ is larger: a biased coin carries more information
• They are equal: $h$ is constant on $(0, 1)$

18. Somewhere new

Read one short sensor log, then answer both questions from it.

This task has no paper form; do it on a device.

19. Lesson test

Lesson test: one question per skill, one attempt each, no hints. Your answers are checked when you submit.

20. Test question

As $p$ runs over $[0, 1]$, what set of values does $h(p)$ take? Give it as an interval.

This task has no paper form; do it on a device.

21. What you can do now

You can evaluate, mirror and compare the binary entropy function. Say in your own words why $h(0.9)$ and $h(0.1)$ must be equal, and why neither can be as large as $h(0.5)$. Next: the bounds every entropy obeys, and the convexity tool that proves them.

Working for the steps left to you

13. Your turn: $h(0.75)$, step 2