{"id":1601,"date":"2019-08-13T17:34:51","date_gmt":"2019-08-13T21:34:51","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/simplestats\/?post_type=chapter&p=1601"},"modified":"2019-08-13T19:03:36","modified_gmt":"2019-08-13T23:03:36","slug":"3-6-outliers","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/simplestats\/chapter\/3-6-outliers\/","title":{"raw":"3.6 Outliers","rendered":"3.6 Outliers"},"content":{"raw":"[latexpage]\r\n\r\nOut of the three measures of central tendency, the mean is the only one that takes into account the actual numerical values of the cases. As such, it is easily affected by the size of the values: a sequence of numbers such as \"1, 5, 7, 10, 15\" will produce a smaller mean than a sequence of numbers like \"100, 50, 75, 130, 90\".\r\n\r\n \r\n\r\nWhen all values to be averaged are of relatively comparable magnitude, the mean does a good job at reflecting the central tendency of a variable -- that is why it is the most familiar and widely used measure. However, when a variable contains an extremely small or an extremely large value (or several values) compared to the rest of the values, the mean gets easily distorted<\/strong> and stops reflecting the central tendency \"truthfully\", as it were. Extremely small and extremely large values are called statistical\u00a0outliers<\/em>.<\/strong>\r\n\r\n \r\n\r\nWhile there is a convenient method for identifying outliers (using a concept\u00a0called\u00a0<\/span>interquartile range\u00a0<\/em>which\u00a0we will discuss in the next chapter<\/span>), at this stage it is not necessary that you be so technical. You can visually identify outliers, albeit less precisely, by the \"disturbance\" in the general pattern of the data you observe. For example, if you have values like \"1, 5, 7, 10, 15\", a value of 130 in that sequence would be considered an outlier. Similarly, if you have values like \"100, 80, 75, 130, 90\", a value of 5 would be an outlier.<\/span>\r\n\r\n \r\n\r\nLet's calculate the means of the two sequences, first with and then without the so-called outliers and see what happens.\r\n\r\n \r\n\r\nThe first sequence is 1, 5, 7, 10, 15 and we want to see what happens when we add 130.\r\n\r\n \r\n\r\n$$\\frac{(1+5+7+10+15)}{5}=\\frac{38}{5}=7.6$$\r\n\r\n \r\n\r\nWe add 130 to the sequence:\r\n\r\n \r\n\r\n$$\\frac{(1+5+7+10+15+130)}{6}=\\frac{168}{6}=28$$\r\n\r\n \r\n\r\nBoth means, 7.6 and 28, are the true averages of the sequences of values as listed. However, the addition of an uncommonly large number \"pulled\" the mean away from the \"centre\" of the original data.\r\n\r\n \r\n\r\nHow truthfully does 28 represent the \"centre\" of a sequence where the majority of the cases's values (in fact, five out of the six values) are 15 and below? Not that much.[footnote]If you believe it's not the magnitude of the value but just its addition that causes the \"pulling\" of the mean, consider redoing the example with adding 18, instead of 130. Then we have $\\frac{(1+5+7+10+15+18)}{6}=\\frac{56}{6}=9.3$. The \"pull\" from 7.6 to 9.3 is much smaller than from 7.6 to 28. The value 9.3 reflects the central tendency of the data more truthfully than 28 does.[\/footnote]\r\n\r\n \r\n\r\nTo demonstrate the effect of an extremely small value, we continuing with the next sequence:\r\n\r\n \r\n\r\n$$\\frac{(100+80+75+130+90)}{5}=\\frac{475}{5}=95$$\r\n\r\n \r\n\r\nAdding a value of 5 to the sequence produces the following:\r\n\r\n \r\n\r\n$$\\frac{(100+80+75+130+90+5)}{6}=\\frac{460}{6}=80$$\r\n\r\n \r\n\r\nSimilarly as with the effect on the mean of the first sequence, the mean here gets \"pulled\", but in the opposite direction, from 95 to 80. Both means are technically true averages of their respective values but the latter one is \"artificially\" low: after all, four out of the six values are the same or higher.[footnote]Again, if we added a value of a comparable size to this sequence instead of 5, the mean would not be impacted as much: $\\frac{(100+80+75+130+90+70)}{6}=\\frac{545}{6}=90.8.$ Consider the \"pull\" from 95 to 80 vs. from 95 to 90.8.[\/footnote]\r\n\r\n \r\n\r\nWhat this tells you is that the mean is an unstable measure of central tendency, prone to being affected by outliers.<\/strong> Contrast this to what you know about the median: the median does not take the magnitude of the values into consideration, beyond their order. Thus, as explained in the previous Section 3.3 (https:\/\/pressbooks.bccampus.ca\/simplestats\/chapter\/3-3-the-median-with-frequency-tables\/), adding a value (be it extremely small or extremely large) to a sequence does not affect the median much -- unlike the mean. The median of 1, 5, 7, 10, 15 is 7 (there are two values above and two below it), and whether we add 130 or 18, it doesn't matter: it's just an additional value in the sequence.[footnote]The median of 1, 5, 7, 10, 15, 18 is between 7 and 10, i.e., 8.5 (since we need the half-way distance between 7 and 10, we use the average of 7 and 10, that is 7+10=17 and divide it by 2 to get 8.5).\u00a0 The median of 1, 5, 7, 10, 15, 130 is exactly the same -- it is still half-way between the two middle values, 7 and 10, or again 8.5.\u00a0[\/footnote]\r\n\r\n \r\n\r\nSince the mean is prone to being affected by outliers, while the median is not, in some situations it is advisable to report the median as a more \"valid\" measure of the typical cases\/\"centre\" of the data rather than the mean.<\/strong>\u00a0Specifically, watch out for reports on average income, average age, average weight, etc. where a few outliers can skew<\/em> a variable's distribution.\r\n\r\n \r\n
\r\n

Watch Out!! #8<\/strong><\/span> ... for Reports on Averages of Variables Prone to Skewing by Outliers<\/em><\/p>\r\n\r\n<\/header>\r\n

\r\n\r\n \r\n\r\nImagine a small company advertising an open position by claiming that the average salary of their employees is 100 thousand dollars per year. For simplicity's sake, let's assume the company has ten employees and these are their salaries:\r\n\r\n<\/div>\r\nTable 3.8 Employee Salaries (Hypothetical Data)\u00a0<\/em>\r\n
\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Value (in thousands)<\/strong><\/td>\r\nFrequency<\/strong><\/td>\r\n<\/tr>\r\n
70<\/td>\r\n5<\/td>\r\n<\/tr>\r\n
87.5<\/td>\r\n4<\/td>\r\n<\/tr>\r\n
300<\/td>\r\n1<\/td>\r\n<\/tr>\r\n
TOTAL<\/strong><\/td>\r\n10<\/strong><\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nYou can check for yourself what the average annual salary is:\r\n\r\n \r\n\r\n$$\\frac{\\sum\\limits_{i=1}^{N}{x_i}}{N}=\\frac{70(5)+87.5(4)+300(1)}{10}=\\frac{350+350+300}{10}= \\frac{1000}{10}=100$$\r\n\r\n \r\n\r\nor, indeed, 100 thousand dollars. However, how representative this annual salary is for the regular employee? After all, nine out of ten employees of the company get less than that. The average annual salary reported is inflated by the very high salary of one employee (perhaps the manager), a clear outlier.\r\n\r\n \r\n\r\nLet's instead look at the median. We start by arrange the values in order:\r\n\r\n \r\n\r\n70, 70, 70, 70, 70, 87.5, 87.5, 87.5, 87.5, 300\r\n\r\n \r\n\r\nUsing the formula for finding the position of the median, we have\r\n\r\n \r\n\r\n$$\\frac{(N+1)}{2}=\\frac{(10+1)}{2}=\\frac{11}{2}=5.5$$\r\n\r\n \r\n\r\nI.e., we find that the median falls between the fifth and the sixth value in the order, or between 70 and 87.5. The halfway point between these two values is found by averaging them:\r\n\r\n \r\n\r\n$$\\frac{(70+87.5)}{2}=\\frac{157.5}{2}=78.75$$\r\n\r\n \r\n\r\nwhich shows us that the median annual salary of the employees in that company is \\$78,750. This is a lot less than the touted average of \\$100,000 and a lot more reflective of what nine out of ten employees receive.\r\n\r\n \r\n\r\n<\/div>\r\n<\/div>\r\n \r\n\r\nExamples like the Watch Out!! #8<\/em> above show that relying on the mean can be tricky, and in some cases can be deliberately used to \"lie with statistics\" (i.e., a report might be technically correct but at the same time very misleading).\u00a0Thus, generally reporting all three central tendency measures is the way to go<\/strong> and you, as a beginner researchers should do that.<\/span>\r\n\r\n \r\n\r\nFinally, you can observe a skew in the data even visually by looking at an interval\/ratio variable's graphical representation, i.e., its histogram. Extremely high values tend to \"pull\" the mean to the right of the \"centre\", i.e., with the majority of cases being relatively smaller, the few high values will produce a \"tail\" on the right side of the distribution (a.k.a. positive skew<\/em>). On the other hand, extremely low values tend to \"pull\" the mean to the left of the \"centre\", i.e., with the majority of cases being relatively larger, the few low values will produce a \"tail\" on the left side of the distribution (a.k.a. negative skew<\/em>).\r\n\r\n \r\n\r\nAs well, since the median indicates the \"centre\" of the data better, a mean smaller than the median would typically indicate a negative\/left skew, while a mean larger than the median would typically indicate a positive\/right skew. When you observe a skew in the data, the median would typically be a the preferred measure of central tendency.<\/span>\r\n\r\n \r\n\r\nObserve the positive skew in Fig. 3.2 below.\r\n\r\n \r\n\r\nFigure 3.1 Number of Cigarettes Smoked Per Day by Occasional Smokers (CCHS 15\/16)<\/em>\r\n\r\n\"\"\r\n\r\nThe reason the numbers on the horizontal axis reach as high as 100 despite the fact that there appears to be nothing there is because there is at least one outlier case -- a respondent who said they were an occasional smoker but reported smoking 99 cigarettes per day.[footnote]Whether this is to be believed is not important here, just the fact that such a value exists in the data. You will learn what is to be done about outliers in statistical analysis in Chapter 4.[\/footnote] Thus the distribution has a long right-side \"tail\", as it were, which you can better see in Fig. 3.2 providing the \"zoomed-in\" version of the histogram above. (The \"tail\" is what you will have if you trace an imaginary line through the tops of all the bars in the histogram down to the single case of 99 cigarettes per day.)\r\n\r\n \r\n\r\nFigure 3.2<\/em>\u00a0Number of Cigarettes Smoked Per Day by Occasional Smokers (CCHS 15\/16), Zoomed<\/em>\r\n\r\n\"\"\r\n\r\nIn this case the median is 3 cigarettes smoked per day by an occasional smoker. The mean is 4.33, and as expected, it is larger than the median.\r\n\r\n \r\n\r\nSimilarly, an exceptionally small value compared to the bulk of the cases will produce a negatively-skewed histogram where the distribution has a \"tail\" but on the left of where most cases are. In that case the mean will be smaller than the median.","rendered":"

Out of the three measures of central tendency, the mean is the only one that takes into account the actual numerical values of the cases. As such, it is easily affected by the size of the values: a sequence of numbers such as “1, 5, 7, 10, 15” will produce a smaller mean than a sequence of numbers like “100, 50, 75, 130, 90”.<\/p>\n

 <\/p>\n

When all values to be averaged are of relatively comparable magnitude, the mean does a good job at reflecting the central tendency of a variable — that is why it is the most familiar and widely used measure. However, when a variable contains an extremely small or an extremely large value (or several values) compared to the rest of the values, the mean gets easily distorted<\/strong> and stops reflecting the central tendency “truthfully”, as it were. Extremely small and extremely large values are called statistical\u00a0outliers<\/em>.<\/strong><\/p>\n

 <\/p>\n

While there is a convenient method for identifying outliers (using a concept\u00a0called\u00a0<\/span>interquartile range\u00a0<\/em>which\u00a0we will discuss in the next chapter<\/span>), at this stage it is not necessary that you be so technical. You can visually identify outliers, albeit less precisely, by the “disturbance” in the general pattern of the data you observe. For example, if you have values like “1, 5, 7, 10, 15”, a value of 130 in that sequence would be considered an outlier. Similarly, if you have values like “100, 80, 75, 130, 90”, a value of 5 would be an outlier.<\/span><\/p>\n

 <\/p>\n

Let’s calculate the means of the two sequences, first with and then without the so-called outliers and see what happens.<\/p>\n

 <\/p>\n

The first sequence is 1, 5, 7, 10, 15 and we want to see what happens when we add 130.<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[\frac{(1+5+7+10+15)}{5}=\frac{38}{5}=7.6\]\"<\/p>\n

 <\/p>\n

We add 130 to the sequence:<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[\frac{(1+5+7+10+15+130)}{6}=\frac{168}{6}=28\]\"<\/p>\n

 <\/p>\n

Both means, 7.6 and 28, are the true averages of the sequences of values as listed. However, the addition of an uncommonly large number “pulled” the mean away from the “centre” of the original data.<\/p>\n

 <\/p>\n

How truthfully does 28 represent the “centre” of a sequence where the majority of the cases’s values (in fact, five out of the six values) are 15 and below? Not that much.[1]<\/sup><\/a><\/p>\n

 <\/p>\n

To demonstrate the effect of an extremely small value, we continuing with the next sequence:<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[\frac{(100+80+75+130+90)}{5}=\frac{475}{5}=95\]\"<\/p>\n

 <\/p>\n

Adding a value of 5 to the sequence produces the following:<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[\frac{(100+80+75+130+90+5)}{6}=\frac{460}{6}=80\]\"<\/p>\n

 <\/p>\n

Similarly as with the effect on the mean of the first sequence, the mean here gets “pulled”, but in the opposite direction, from 95 to 80. Both means are technically true averages of their respective values but the latter one is “artificially” low: after all, four out of the six values are the same or higher.[2]<\/sup><\/a><\/p>\n

 <\/p>\n

What this tells you is that the mean is an unstable measure of central tendency, prone to being affected by outliers.<\/strong> Contrast this to what you know about the median: the median does not take the magnitude of the values into consideration, beyond their order. Thus, as explained in the previous Section 3.3 (https:\/\/pressbooks.bccampus.ca\/simplestats\/chapter\/3-3-the-median-with-frequency-tables\/), adding a value (be it extremely small or extremely large) to a sequence does not affect the median much — unlike the mean. The median of 1, 5, 7, 10, 15 is 7 (there are two values above and two below it), and whether we add 130 or 18, it doesn’t matter: it’s just an additional value in the sequence.[3]<\/sup><\/a><\/p>\n

 <\/p>\n

Since the mean is prone to being affected by outliers, while the median is not, in some situations it is advisable to report the median as a more “valid” measure of the typical cases\/”centre” of the data rather than the mean.<\/strong>\u00a0Specifically, watch out for reports on average income, average age, average weight, etc. where a few outliers can skew<\/em> a variable’s distribution.<\/p>\n

 <\/p>\n

\n
\n

Watch Out!! #8<\/strong><\/span> … for Reports on Averages of Variables Prone to Skewing by Outliers<\/em><\/p>\n<\/header>\n

\n

 <\/p>\n

Imagine a small company advertising an open position by claiming that the average salary of their employees is 100 thousand dollars per year. For simplicity’s sake, let’s assume the company has ten employees and these are their salaries:<\/p>\n<\/div>\n

Table 3.8 Employee Salaries (Hypothetical Data)\u00a0<\/em><\/p>\n

\n\n\n\n\n\n\n\n
Value (in thousands)<\/strong><\/td>\nFrequency<\/strong><\/td>\n<\/tr>\n
70<\/td>\n5<\/td>\n<\/tr>\n
87.5<\/td>\n4<\/td>\n<\/tr>\n
300<\/td>\n1<\/td>\n<\/tr>\n
TOTAL<\/strong><\/td>\n10<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

You can check for yourself what the average annual salary is:<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[\frac{\sum\limits_{i=1}^{N}{x_i}}{N}=\frac{70(5)+87.5(4)+300(1)}{10}=\frac{350+350+300}{10}= \frac{1000}{10}=100\]\"<\/p>\n

 <\/p>\n

or, indeed, 100 thousand dollars. However, how representative this annual salary is for the regular employee? After all, nine out of ten employees of the company get less than that. The average annual salary reported is inflated by the very high salary of one employee (perhaps the manager), a clear outlier.<\/p>\n

 <\/p>\n

Let’s instead look at the median. We start by arrange the values in order:<\/p>\n

 <\/p>\n

70, 70, 70, 70, 70, 87.5, 87.5, 87.5, 87.5, 300<\/p>\n

 <\/p>\n

Using the formula for finding the position of the median, we have<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[\frac{(N+1)}{2}=\frac{(10+1)}{2}=\frac{11}{2}=5.5\]\"<\/p>\n

 <\/p>\n

I.e., we find that the median falls between the fifth and the sixth value in the order, or between 70 and 87.5. The halfway point between these two values is found by averaging them:<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[\frac{(70+87.5)}{2}=\frac{157.5}{2}=78.75\]\"<\/p>\n

 <\/p>\n

which shows us that the median annual salary of the employees in that company is $78,750. This is a lot less than the touted average of $100,000 and a lot more reflective of what nine out of ten employees receive.<\/p>\n

 <\/p>\n<\/div>\n<\/div>\n

 <\/p>\n

Examples like the Watch Out!! #8<\/em> above show that relying on the mean can be tricky, and in some cases can be deliberately used to “lie with statistics” (i.e., a report might be technically correct but at the same time very misleading).\u00a0Thus, generally reporting all three central tendency measures is the way to go<\/strong> and you, as a beginner researchers should do that.<\/span><\/p>\n

 <\/p>\n

Finally, you can observe a skew in the data even visually by looking at an interval\/ratio variable’s graphical representation, i.e., its histogram. Extremely high values tend to “pull” the mean to the right of the “centre”, i.e., with the majority of cases being relatively smaller, the few high values will produce a “tail” on the right side of the distribution (a.k.a. positive skew<\/em>). On the other hand, extremely low values tend to “pull” the mean to the left of the “centre”, i.e., with the majority of cases being relatively larger, the few low values will produce a “tail” on the left side of the distribution (a.k.a. negative skew<\/em>).<\/p>\n

 <\/p>\n

As well, since the median indicates the “centre” of the data better, a mean smaller than the median would typically indicate a negative\/left skew, while a mean larger than the median would typically indicate a positive\/right skew. When you observe a skew in the data, the median would typically be a the preferred measure of central tendency.<\/span><\/p>\n

 <\/p>\n

Observe the positive skew in Fig. 3.2 below.<\/p>\n

 <\/p>\n

Figure 3.1 Number of Cigarettes Smoked Per Day by Occasional Smokers (CCHS 15\/16)<\/em><\/p>\n

\"\"<\/p>\n

The reason the numbers on the horizontal axis reach as high as 100 despite the fact that there appears to be nothing there is because there is at least one outlier case — a respondent who said they were an occasional smoker but reported smoking 99 cigarettes per day.[4]<\/sup><\/a> Thus the distribution has a long right-side “tail”, as it were, which you can better see in Fig. 3.2 providing the “zoomed-in” version of the histogram above. (The “tail” is what you will have if you trace an imaginary line through the tops of all the bars in the histogram down to the single case of 99 cigarettes per day.)<\/p>\n

 <\/p>\n

Figure 3.2<\/em>\u00a0Number of Cigarettes Smoked Per Day by Occasional Smokers (CCHS 15\/16), Zoomed<\/em><\/p>\n

\"\"<\/p>\n

In this case the median is 3 cigarettes smoked per day by an occasional smoker. The mean is 4.33, and as expected, it is larger than the median.<\/p>\n

 <\/p>\n

Similarly, an exceptionally small value compared to the bulk of the cases will produce a negatively-skewed histogram where the distribution has a “tail” but on the left of where most cases are. In that case the mean will be smaller than the median.<\/p>\n

  1. If you believe it's not the magnitude of the value but just its addition that causes the \"pulling\" of the mean, consider redoing the example with adding 18, instead of 130. Then we have \"\frac{(1+5+7+10+15+18)}{6}=\frac{56}{6}=9.3\". The \"pull\" from 7.6 to 9.3 is much smaller than from 7.6 to 28. The value 9.3 reflects the central tendency of the data more truthfully than 28 does. ↵<\/a><\/li>
  2. Again, if we added a value of a comparable size to this sequence instead of 5, the mean would not be impacted as much: \"\frac{(100+80+75+130+90+70)}{6}=\frac{545}{6}=90.8.\" Consider the \"pull\" from 95 to 80 vs. from 95 to 90.8. ↵<\/a><\/li>
  3. The median of 1, 5, 7, 10, 15, 18 is between 7 and 10, i.e., 8.5 (since we need the half-way distance between 7 and 10, we use the average of 7 and 10, that is 7+10=17 and divide it by 2 to get 8.5).\u00a0 The median of 1, 5, 7, 10, 15, 130 is exactly the same -- it is still half-way between the two middle values, 7 and 10, or again 8.5.\u00a0 ↵<\/a><\/li>
  4. Whether this is to be believed is not important here, just the fact that such a value exists in the data. You will learn what is to be done about outliers in statistical analysis in Chapter 4. ↵<\/a><\/li><\/ol><\/div>","protected":false},"author":533,"menu_order":6,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"class_list":["post-1601","chapter","type-chapter","status-publish","hentry"],"part":24,"_links":{"self":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters\/1601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/users\/533"}],"version-history":[{"count":10,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters\/1601\/revisions"}],"predecessor-version":[{"id":1616,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters\/1601\/revisions\/1616"}],"part":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/parts\/24"}],"metadata":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters\/1601\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/media?parent=1601"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapter-type?post=1601"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/contributor?post=1601"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/license?post=1601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}