6.7.1 Additional Confidence Intervals Considerations

Precision vs. certainty. One thing you you might have noticed from the calculations in the examples in the previous section is that the more certainty you get, the larger your confidence interval becomes (or vice versa: the smaller the interval, the less precise your estimate):

Based on the annual income details from Example 6.4, we had

• between $49,700 and $50,300 with 68% confidence;
• between $49,505 and $50,495 with 90% confidence;
• between $49,412 and $50,588 with 95% confidence; and
• between $49,226 and $50,774 with 99% confidence.

Of course, who wouldn’t want both more precise and more certain estimates? Unfortunately there simply is no way to have our cake and eat it too: As you can see above, the more confident in our estimate we get, the more the error bounds of the confidence intervals spread out wider. There is a trade-off between precision and confidence. The more precise our estimate, the less certain we are of it; the more confident we are in our estimate, the less precise our “guess” is.

Logically, this makes a lot of sense: imagine the population parameter as a target and estimation as throwing a dart at it. The smaller the target, the more precise you’ll have to be but also the less confident of hitting it. At the same time, increasing the target size will accommodate less precise “shots” while simultaneously increasing the certainty of the target being hit.

And why can’t we have a 100% CI? The non-technical answer is simply because a statistical estimator is based on a sample drawn from a population of interest: as long as you don’t have data from your entire population, there will always be a possibility for random error (and uncertainty).

The more technical answer lies in the characteristics of the normal probability distribution. Specifically, the normal curve approaches but never reaches the horizontal axis; the probability in its “tails” is not bound — i.e., a probability for any z-value exists, no matter how small or large, and it never reaches 0. Thus, a 100% confidence interval would result in -∞ to +∞ , i.e., it would be virtually infinitely large, to accommodate the perfect certainty. Logically, no bound, finite interval can provide 100% certainty by the nature of statistical inference itself. (Since, at 100%, it would stop being inference altogether: we will have no need to estimate, as we would know.)

The effect of sample size on confidence intervals. Let’s also consider the effect of sample size on the precision and level of certainty of confidence intervals. In Section 6.5 (https://pressbooks.bccampus.ca/simplestats/chapter/6-5-the-sampling-distribution/)  I attempted to convince you that increasing the sample size beyond a specific (large) number becomes not only unfeasible in a world of limited resources but also statistically pointless. Let’s see if I could further support my claim by the effect of sample size on the standard error.

To recall, we find the standard error in the following way:

where we estimate σ (the standard deviation of the population) with s (the standard deviation of the sample) to get

We already established that a larger N would result in a smaller standard error (as N is in the denominator). Given the formula for calculating confidence intervals, a smaller standard error should in turn lead to smaller intervals (i.e., to more precise estimates) at a fixed level of certainty. The question is — how much smaller?

Most people would agree that $49,412 to $50,588 is precise enough, and that there’s no need to waste six times more resources on such a relatively insignificant gain in precision when it comes to average annual income^[1].

Bear in mind, however, that had we been discussing effectiveness of a life-saving medical treatment instead of average annual income, our preferences regarding the trade-off between precision and cost would most likely be different. Thus, the actual value of increasing sample size cannot be judged solely on statistics grounds: what is considered a small/insignificant change in precision for one thing may very well be a large and worthy change in another context. Still, in social science research there’s rarely a need for such increasing precision of inference no matter the costs, even if larger samples are generally preferred^[2].