What Is The Margin of Error?

Header image by @jeswinthomas via Unsplash

If you've ever dabbled in the world of statistics, you might've heard a whisper or two about the margin of error, sometimes abbreviated as MoE. But what does it mean, and why does it matter?

So what does it mean?

Imagine you're baking the world's biggest cookie, but you only have a tiny spoon to sample the dough. Will your spoonful taste the same as the whole batch? Maybe... maybe not. That's the mystery the MoE tries to solve - that is, how much does a sample represent a whole.

Created with Midjourney | Prompt: “the world's biggest chocolate chip cookie, illustration, vector --v 5.2 --s 750”

Why is the margin of error important?

MoE helps keep our overconfidence in check, and it helps us avoid over-generalizing the findings of a survey based on the sample size you're working with. In simple terms, MoE is a tool for measuring the potential deviation from reality when sampling from a larger population.

Generally speaking, the more samples you have in a dataset, the lower the MoE. And the lower the MoE is, the less deviation from reality we can expect.

Here's a quick example. Let's say you ran a nationally representative survey of 1,000 Canadians and found that 60% of the respondents prefer chocolate chip cookies to raisin cookies. Does this 60% represent the views of all Canadians? After all, the population of Canada (at the time of writing) is 38 million, which means a poll of 1,000 people only includes 0.00003% of the actual population. So can we really use the 60% result in our poll to generalize about 38 million people?

This is where MoE comes in. It tells you how much deviation from your sample result you can expect if you were to actually interview the entire population (e.g. 38 million Canadians) or run the poll 100 times.

Let's assume we know our margin of error is 3%. This means that if our poll showed that 60% of Canadians prefer chocolate chip cookies, then we can expect that between 57% and 63% (i.e. +/- 3%) would answer this way if we polled the entire population.

How does the margin of error work?

The equation for MoE looks something like this:

But to really understand the equation, we need to unpack a few key concepts, including sample proportion (p), sample size (n), Z-score (Z), and confidence level.

Sample Proportion (p)

To calculate MoE, you first need to find the proportion (or percentage) of the sample that picked a particular answer. For example, if you polled 1,000 people on their preference for chocolate chip cookies and 600 said "yes," your sample proportion (p) would be 600/1000 = 0.6 (or 60%).

Sample Size (n)

This is the total number of observations in your sample. Using the example above, this would be n=1,000.

Z-score (Z) and Confidence Level

A Z-score is a way of understanding how far away a particular data point is from the average in a dataset. Imagine you're a student, and you scored 85 on a test. And let's say you knew the class average was 75 and the standard deviation (which measures how spread out the scores are) was 10.

This is where Z-scores come in handy. A Z-score of 1 would mean your score is one standard deviation above the average. A Z-score of -1 would mean your score is one standard deviation below the average.

So, in our example, your Z-score would be calculated as follows:

(your score - average score) / standard deviation = (85 - 75) / 10 = 1

This means your score was one standard deviation above the class average.

On the other hand, the confidence level corresponds to how sure you want to be about your estimates. There's no calculation to be done here; you simply choose the confidence level you wish to use. Most of the time, you would go with 95%, but other common choices are 90% and 99%.

And for each confidence level, there's a corresponding Z-score, which you can see below:

• At 90% confidence, the z-score is 1.645

• At 95% confidence, the z-score is 1.96

• At 99% confidence, the z-score is 2.575

The good news is that if you're manually calculating MoE, you don't have to calculate the Z-Score every time, as you can just refer to the figures above depending on the confidence level you want to you.

Now that we have all the pieces let's try calculating MoE, assuming we have the following info:

• A survey with 1,000 samples (n)

• A result where 60% of respondents stated they prefer chocolate chip cookies (p)

• We'll use a 95% confidence level, which means we have a z-score of 1.96 (z)

Using the above info, we can plug this into our equation as follows.

1.96 * √((0.6 * (1 – 0.6) / 1000)

And this will give us a margin of error of 3.036%.

Using a margin of error calculator

If this all sounds a bit complicated, fret not because you can use a margin of error calculator to simplify this process. Typically, when using an MoE calculator, you'll need the following three pieces of information.

• Sample size (n)

• Population Size (N) - This is the total population you are studying. For example, if you were studying the Canadian population, this would be 38,250,000. Population Pyramid is a great resource if you need to get population sizes by country, broken down by age and gender.

• Confidence level

• Sample Proportion - It's worth noting that not all MoE calculators require this. SurveyMonkey's calculator, for example, only requires the sample size, population size and confidence level.

You can use an MoE calculator at the early stage of a research project to determine how many samples you should collect. Or you can use it after collecting your data to assess the MoE based on what you have.

I have my own margin of error calculator on this site, which you can try by clicking the button below.

Some limitations with the margin of error

So we've learned that MoE can be pretty helpful in preventing the over-generalization of a survey finding. But it's important to remember that MoE doesn't factor in your sample's underlying quality or composition. Some data sources can have an inherent bias that skews your results.

For example, if Fox News ran a poll on its website, we probably wouldn't assume the poll results were representative of the opinions of all Americans, right?

This is because Fox News, like many news media organizations, has an audience with a particular demographic and/or psychographic orientation that doesn't reflect the larger population. Fox News' audience skews toward a conservative ideology, while CNN's toward a liberal ideology. On the other hand, Bloomberg's audience tends to skew older (35 and above) and to people working in white-collar, professional vocations, while BuzzFeed's audience skews younger.

Each of these media organizations has audiences with particular interests and tastes, and running a poll on any of their websites wouldn't yield results that reflect the opinions of all Americans.

And this is where professional panel companies come into the picture. In the world of online quantitative research (aka polling), the way in which we source the samples, manage fieldwork, and clean data is essential to building what we call "representative data." Most online sample suppliers, known as panel providers, have rigorous processes in place to detect survey fraud and attention levels, and they use tactics, such as the use of interlocking demographic quotas, to ensure the results are representative of a population. So in summary, professional research firms such as panel providers can help address potential issues of quality or bias that one may encounter from a particular data source.

But the important takeaway here is that the margin of error can’t fix a problematic sample. If steps weren’t taken to ensure the underlying sample is representative, then calculating the margin of error would be pointless.

Conclusion

The margin of error, or MoE for short, is a useful statistical tool to shield against overconfidence in a dataset. It's there to guide you and, most importantly, to remind us all that when it comes to understanding our complex world through data, sometimes it's okay to be a little unsure.