Confidence Interval of a Survey

A confidence interval is a lower and upper bound that defines a margin of error for the raw results of a survey. The confidence interval evaluates the quality and accuracy of the estimate obtained with the sample surveyed.

This interval applies to all types of survey / sampling (street interviews, online surveys) in order to evaluate the trust score that can be attributed to it.

The interval depends mainly on the number of people questioned, if a poll has several questions and the number of answers obtained differs for each question, then the confidence interval must be calculated for each question. Conversely, if all the questions have the same number of respondents, then the confidence interval is the same for all the questions, and therefore for the survey as a whole.

The frequency observed in a sample is noted $ f $ and the probability in the total population is noted $ p $, these values are often confused.

For a survey of $ N $ people resulting in the frequency $ f $ and the probability $ p $, then the 95% confidence interval is $$ \left[p-1.96\frac{\sqrt{f(1-p)}}{\sqrt n},p+1.96\frac{\sqrt{p(1-p)}}{\sqrt n}\right] $$

With 1.96 the value of the 2.5 percentile of the normal distribution (for 99%, the value will be 2.58).

The reliability of a survey (and its interpretation) depends on several key factors:

— Selection methodology: A representative sample of the target population must be selected (randomly).

— Sample size: The greater the number of participants, the smaller the confidence interval.

— Wording of questions: Unclear or clumsy questions can lead to distorted results.

— Response rate: respondents may differ greatly from non-respondents (solicited in the sample).

— Statistical analysis: the tools and methods or even errors in calculation or interpretation can affect the validity of the conclusions.

— Objectivity of the sponsor: potential conflicts of interest may influence all or part of the survey.

— Temporality: polls only provide a snapshot of information that can change over time.

In order to reduce/improve a confidence interval, it is necessary to increase $ N $ (the number of items or people participating in the survey).

The calculation of the confidence interval is not conditioned on the total number of people in the population.

When the sample size $ n $ is very large compared to the population size $ N $, it is possible to use a finite population correction factor (FPC) to adjust the formula of the margin of error.

$$ FPC = \sqrt{\frac{N-n}{N-1}} $$

By multiplying the margin of error by this factor, it is adjusted, especially for a large sampling fraction, because the correction factor reduces it.

The value 1.96 is approximate, but generally sufficient for the majority of applied calculations, however, a more precise value would be 1.95996 (5 digits) or 1.9599639845 (10 digits). Ditto for 2.58 which is a rounding for 2.57583 (5 digits) or 2.5758293035 (10 digits).

The exact value is $ \sqrt{2} \operatorname{erf}^{-1}(95/100) $ for 95%, with inverse error function $ \operatorname{erf}^{-1}(\operatorname{erf}(x)) = x $ and $ \operatorname{erf}(x) $ the error function.

When the probability is close to 0, the calculation of the confidence interval can lead to probabilities outside the interval $ [0,1] $ which is impossible. One rule is to use the limit as $ 3 / N $.

The confidence interval applies to each survey question/measure.

It is then possible to define the overall confidence interval of the survey (generally an average of the confidence intervals of each question).

But if all the questions are answered by exactly the same people, then all the questions will have the same confidence interval and the overall confidence interval will also be the same.