Back to the on-screen lesson ·
Whether an estimator aims at the parameter, why the sample variance divides by one less than the sample size, and what more data can and cannot fix.
Paper packet. Every task here also exists on screen, where it is checked automatically; answers written on paper are not assessed by Nydus. When you are back at a device, enter your answers there.
You will compute the bias of an estimator from its expectation, show that the sample mean is unbiased, explain why the sample variance divides by one less than the sample size, and tell a fault in where an estimator aims from a fault in how far it scatters.
The last lesson showed that an estimator is a random variable with a distribution of its own. This one asks the first question anybody asks of a distribution: where is its centre, and is that centre the right place? The only tool needed is the linearity of expectation.
The bias of $\hat\theta$ is $E[\hat\theta] - \theta$ — the expectation minus the parameter, in that order, so a negative bias means the estimator runs low. An estimator with bias zero for every value of the parameter is unbiased. Degrees of freedom counts the independent pieces of information left after the quantities estimated from the data have been paid for.
An estimator $\hat\theta$ is unbiased for $\theta$ when
$$E[\hat\theta] = \theta \quad\text{for every value of } \theta,$$
and its bias is $E[\hat\theta] - \theta$ otherwise. Note what this is and is not a statement about: it is about the sampling distribution, not about the sample that arrived. A single estimate is never biased; the recipe that produced it may be.
The sample mean is unbiased for $\mu$ because expectation is linear:
$$E[\bar X] = \frac{1}{n}\sum_{i=1}^{n} E[X_i] = \frac{n\mu}{n} = \mu.$$
The sample variance is the case worth knowing in detail. Deviations are measured from $\bar X$, which is by construction the point the data sit closest to, so $\sum (X_i - \bar X)^2$ comes out smaller than it would about the true mean. Exactly one observation's worth smaller:
$$E\left[\sum_{i=1}^{n}(X_i - \bar X)^2\right] = (n - 1)\sigma^{2}.$$
Dividing by $n$ therefore gives an estimator with expectation $\frac{n-1}{n}\sigma^{2}$ — biased low by $\sigma^{2}/n$ — and dividing by $n - 1$ gives an unbiased one. The correction counts degrees of freedom: one was spent estimating the mean, and the divisor is what is left.
Unbiasedness is worth having and is not worth everything. It does not survive a non-linear transformation — an unbiased estimator of $\sigma^{2}$ does not give an unbiased estimator of $\sigma$ — and an unbiased estimator with a huge variance can be worse than a slightly biased one with a small variance. The next lesson puts a number on that trade.
Another way: picture
Three archers shooting at a target. The first puts every arrow in a tight cluster a hand's width to the left: biased, and no number of arrows moves the cluster. The second scatters arrows all round the bullseye: unbiased, and wide. The third is tight on the bullseye. Bias is where the cluster sits; variance is how big it is.
Another way: steps
Two observations, $X_1$ and $X_2$, from a population with variance $\sigma^{2}$. The squared deviation from their own mean is
$$(X_1 - \bar X)^2 + (X_2 - \bar X)^2 = \tfrac{1}{2}(X_1 - X_2)^2,$$
and $X_1 - X_2$ has variance $2\sigma^{2}$ and mean zero, so the expectation of that sum is $\sigma^{2}$ — one $\sigma^{2}$, not two. Dividing by $2$ would give $\sigma^{2}/2$, half the truth; dividing by $1$ gives the truth. With two observations the correction is not a detail of a percent or two: it is a factor of two, and it is visible in a single line of algebra.
Calling one estimate biased. Bias is a property of the recipe across all samples. The number in front of you is high or low, and nobody knows which.
Subtracting the wrong way round. Bias is expectation minus parameter. The other order gives the right size and the wrong sign, and the sign is the useful half.
Believing more data removes a bias. It removes variance. A biased sampling scheme surveyed ten times over is biased ten times over, and the interval around the wrong answer merely gets narrower.
Expecting unbiasedness to survive a square root. It does not. An unbiased estimator of the variance gives a slightly low estimator of the standard deviation, and that is unavoidable rather than a mistake.
$E[\bar X] = \frac{1}{n}\sum E[X_i]$ by linearity, whatever the population is.
No independence is even needed here.
Each $E[X_i] = \mu$, so the sum is $n\mu$ and the expectation is $\mu$.
Unbiased, for every $n$.
A sample of $5$ from a population with $\sigma^{2} = 20$; the sum of squared deviations has expectation $4 \times 20 = 80$.
One less than five.
Dividing by $5$ gives expectation $16$: biased low by $4$.
Twenty per cent short.
Dividing by $4$ gives expectation $20$, which is the parameter.
Unbiased.
Its expectation is half the expectation of the sample mean, so $15$.
Linearity again.
The bias is $15 - 30$.
Expectation minus parameter.
So the bias is $-15$: it runs low by half the parameter, at every sample size.
A population has variance $42$ and a sample of $7$ is drawn. The sum of squared deviations from the sample mean has expectation $252$. For each divisor, give the expectation of the resulting estimator and its bias.
| Expectation | Bias | |
|---|---|---|
| Divided by the sample size | ||
| Divided by one less than the sample size |
An estimator of $\theta$ has expectation $14$ when the true value is $\theta = 22$. What is its bias?
Answer:
A sample of $57$ observations is drawn and an estimator is proposed. Put the steps of showing that it is unbiased into the order they are carried out.
Number the steps in order (write the number in the box):
The sample mean is unbiased for a population mean of $47$. An analyst proposes adding $9$ to it before reporting. Give the expectation of the proposed estimator, and its bias.
Expectation: e. Bias: c.
A report on a survey of $52$ households makes four criticisms. Mark the two that describe a bias rather than a variance.
This task has no paper form; do it on a device.
Lesson test: one question per skill, one attempt each, no hints. Your answers are checked when you submit.
A sample of $14$ observations is used to estimate the population variance, and the population mean is unknown. The sum of squared deviations from the sample mean is divided by what, to give an unbiased estimate?
You can compute a bias, name the degrees of freedom a correction counts, and say which faults in an estimate more data will cure. Say in your own words why a single estimate can never be called biased.
9. Your turn: an estimator that halves the sample mean, when the population mean is $30$, step 3