{"id":135,"date":"2018-10-31T18:13:37","date_gmt":"2018-10-31T22:13:37","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/simplestats\/?post_type=chapter&p=135"},"modified":"2019-11-15T18:52:37","modified_gmt":"2019-11-15T23:52:37","slug":"10-2-1-the-linear-regression-model","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/simplestats\/chapter\/10-2-1-the-linear-regression-model\/","title":{"raw":"10.2.1 The Linear Regression Model and the Line of Best Fit","rendered":"10.2.1 The Linear Regression Model and the Line of Best Fit"},"content":{"raw":"[latexpage]\r\n\r\nYou might have noticed that there was no uncertainty of any kind in the Example 10.2 about the assignment requirements and mark in the previous section. The line in that case represented a deterministic<\/em> relationship -- x<\/em> fully determined\u00a0y<\/em> (i.e., x<\/em> fully explained the variability of y<\/em>) -- hence all the observation were on the line itself.\r\n\r\n \r\n\r\nAs such, this was not a typical situation and this was not a typical regression<\/em> line. In reality, in statistical inference we deal with probabilistic<\/em> associations, where the regression line does not<\/em> capture all observations in itself but their general<\/em> (on average) trend<\/em>. That is, in a usual regression model situation, some observations will be above the line and some below it; thus some observations would be underestimated<\/em> and others would be overestimated<\/em> because the line serves as a prediction<\/em> <\/strong>(an expectation, a summary, a trend) of the association. And as we know by now, predictions\/estimations always contain a level of uncertainty.\r\n\r\n \r\n\r\nSpecifically, we cannot expect that a single independent variable x<\/em> will explain away all<\/em> variability in a dependent variable y<\/em>; there will always be some unexplained (by the regression model) variability left. Figure 10.2 illustrates.\r\n\r\n \r\n\r\nFigure 10.2 Assignment Mark as a Function of Completed Requirements (With Variance)<\/em>\r\n\r\n\"\"\r\n\r\nIn Figure 10.2 I have added seven more observations to the case we had in Figure 10.1 in the previous section, this time allowing for additional variability in the assignment marks: no longer is it enough to know the number of requirements completed to predict the assignment grade. (Imagine that the professor has started evaluating the completed requirements substantively, not just counting them: in this case while the number of requirements is still essential for the grade, something else<\/em>[footnote]This something else<\/em> is an 'unobserved variable', or a variable not included in the model (even though we could speculate about it). This type of unobserved variable\/s is the source for the unexplained variance in y<\/em>.[\/footnote] also affects the final assignment mark.)\r\n\r\n \r\n\r\nAn actual regression model accommodates the uncertainty inherent in estimation through two interrelated concepts, error of prediction<\/em> (a.k.a. statistical error) and residuals<\/em>.<\/strong>\r\n\r\n \r\n\r\nThe error of prediction<\/em> reflects the difference between the observations and the predicted values we would have if we had data about the population.<\/strong> That is, if we imagined a line of best fit of the population[footnote]This line of course does not exist, it is a heuristic device.[\/footnote],\u00a0\u03b1+\u03b2x<\/em>, the difference between our observations and that line would be:\r\n\r\n \r\n\r\n$$y-(\\alpha+\\beta x)=\\epsilon$$ = error of prediction<\/em>[footnote]This is the small-case Greek letter e<\/em>, \u03b5<\/em> [EHpsilon].[\/footnote]\r\n\r\n \r\n\r\nThat is, we need to include the error term in the regression model:\r\n\r\n \r\n\r\n$$y=\\alpa+\\beta x +\\epsilon$$\r\n\r\n \r\n\r\nConsidering that we pretty much never have information about the population, however, we can restate the sample<\/em> regression model like this<\/strong>:\r\n\r\n \r\n\r\n$$y=a+bx+e$$\r\n\r\n \r\n\r\nwhere a<\/em> is the estimated\u00a0\u03b1<\/em>, b<\/em> is the estimated\u00a0\u03b2<\/em>, and e<\/em> is the estimated\u00a0\u03b5<\/em>, with all estimations based on sample data. Note that e<\/em> here is called the residual<\/em>, and it is not only the estimation of the unobservable error of prediction, but also simply the difference between an observation and its predicted value<\/strong>:\r\n\r\n \r\n\r\n$$y-(a+bx)=e$$ = residual<\/em>\r\n\r\n \r\n\r\nSince a+bx<\/em> is the regression line, or the prediction, it also stands for the predicted (estimated values), which we can, as usual, denote $\\hat{y}$. Then, since\r\n\r\n \r\n\r\n$$\\hat{y}=a+bx$$,\r\n\r\n \r\n\r\nwe also have\r\n\r\n \r\n\r\n$$y-\\hat{y}=e$$\r\n\r\n \r\n\r\nor, again, that the residuals are the difference between the observations and their predicted values.<\/strong>\r\n\r\n \r\n\r\nWith this, we come at a full circle and the reason for all the notation and protracted explanations above (and here you thought I was subjecting you to all these equations without a purpose): in a graph, the residuals are simply the distance between the observations and the regression line<\/strong>. (In Figure 10.2 this is the empty space -- the shortest distance -- between an observation and the regression line.)\r\n\r\n \r\n\r\nA comprehensive treatment of the residuals (through a full-blown analysis of variance) is beyond the scope of this book but they do help us understand the nature of the regression line and of the logic of regression in general. You see, the regression line is called a line of best fit<\/em> precisely because it minimizes<\/em> the residuals<\/strong> -- it is created in such a way as to minimize the residuals (and therefore the error of prediction) and fit the data\/observations as best as possible. Visually, this will mean that the line is drawn to pass as close as possible<\/em> to all the observations.\r\n\r\n \r\n\r\nIn fact, linear regression is also called OLS regression<\/em>, which stands for ordinary least squares<\/em>.<\/strong> The least squares<\/em>\u00a0concept comes from the fact that to minimize the distances of the observations to the prediction line, we need to first square them before adding them together[footnote]I.e., $\\Sigma{(y-\\hat{y})^2}$.[\/footnote] -- just like we needed to do that in the calculation of the variance and the sum of squares (or the distances would cancel each other out)[footnote]The ordinary<\/em> part is there to differentiate between another regression version called generalized least squares regression<\/em>, or GLS<\/em> regression (not discussed here).[\/footnote].\r\n\r\n \r\n\r\nBut how do we ensure that the regression line minimizes the residuals? The next section explains.","rendered":"

You might have noticed that there was no uncertainty of any kind in the Example 10.2 about the assignment requirements and mark in the previous section. The line in that case represented a deterministic<\/em> relationship — x<\/em> fully determined\u00a0y<\/em> (i.e., x<\/em> fully explained the variability of y<\/em>) — hence all the observation were on the line itself.<\/p>\n

 <\/p>\n

As such, this was not a typical situation and this was not a typical regression<\/em> line. In reality, in statistical inference we deal with probabilistic<\/em> associations, where the regression line does not<\/em> capture all observations in itself but their general<\/em> (on average) trend<\/em>. That is, in a usual regression model situation, some observations will be above the line and some below it; thus some observations would be underestimated<\/em> and others would be overestimated<\/em> because the line serves as a prediction<\/em> <\/strong>(an expectation, a summary, a trend) of the association. And as we know by now, predictions\/estimations always contain a level of uncertainty.<\/p>\n

 <\/p>\n

Specifically, we cannot expect that a single independent variable x<\/em> will explain away all<\/em> variability in a dependent variable y<\/em>; there will always be some unexplained (by the regression model) variability left. Figure 10.2 illustrates.<\/p>\n

 <\/p>\n

Figure 10.2 Assignment Mark as a Function of Completed Requirements (With Variance)<\/em><\/p>\n

\"\"<\/p>\n

In Figure 10.2 I have added seven more observations to the case we had in Figure 10.1 in the previous section, this time allowing for additional variability in the assignment marks: no longer is it enough to know the number of requirements completed to predict the assignment grade. (Imagine that the professor has started evaluating the completed requirements substantively, not just counting them: in this case while the number of requirements is still essential for the grade, something else<\/em>[1]<\/sup><\/a> also affects the final assignment mark.)<\/p>\n

 <\/p>\n

An actual regression model accommodates the uncertainty inherent in estimation through two interrelated concepts, error of prediction<\/em> (a.k.a. statistical error) and residuals<\/em>.<\/strong><\/p>\n

 <\/p>\n

The error of prediction<\/em> reflects the difference between the observations and the predicted values we would have if we had data about the population.<\/strong> That is, if we imagined a line of best fit of the population[2]<\/sup><\/a>,\u00a0\u03b1+\u03b2x<\/em>, the difference between our observations and that line would be:<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[y-(\alpha+\beta x)=\epsilon\]\"<\/p>\n

= error of prediction<\/em>[3]<\/sup><\/a><\/p>\n

 <\/p>\n

That is, we need to include the error term in the regression model:<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[y=\alpa+\beta x +\epsilon\]\"<\/p>\n

 <\/p>\n

Considering that we pretty much never have information about the population, however, we can restate the sample<\/em> regression model like this<\/strong>:<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[y=a+bx+e\]\"<\/p>\n

 <\/p>\n

where a<\/em> is the estimated\u00a0\u03b1<\/em>, b<\/em> is the estimated\u00a0\u03b2<\/em>, and e<\/em> is the estimated\u00a0\u03b5<\/em>, with all estimations based on sample data. Note that e<\/em> here is called the residual<\/em>, and it is not only the estimation of the unobservable error of prediction, but also simply the difference between an observation and its predicted value<\/strong>:<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[y-(a+bx)=e\]\"<\/p>\n

= residual<\/em><\/p>\n

 <\/p>\n

Since a+bx<\/em> is the regression line, or the prediction, it also stands for the predicted (estimated values), which we can, as usual, denote \"\hat{y}\". Then, since<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[\hat{y}=a+bx\]\"<\/p>\n

,<\/p>\n

 <\/p>\n

we also have<\/p>\n

 <\/p>\n

  <\/span>   <\/span>\"\[y-\hat{y}=e\]\"<\/p>\n

 <\/p>\n

or, again, that the residuals are the difference between the observations and their predicted values.<\/strong><\/p>\n

 <\/p>\n

With this, we come at a full circle and the reason for all the notation and protracted explanations above (and here you thought I was subjecting you to all these equations without a purpose): in a graph, the residuals are simply the distance between the observations and the regression line<\/strong>. (In Figure 10.2 this is the empty space — the shortest distance — between an observation and the regression line.)<\/p>\n

 <\/p>\n

A comprehensive treatment of the residuals (through a full-blown analysis of variance) is beyond the scope of this book but they do help us understand the nature of the regression line and of the logic of regression in general. You see, the regression line is called a line of best fit<\/em> precisely because it minimizes<\/em> the residuals<\/strong> — it is created in such a way as to minimize the residuals (and therefore the error of prediction) and fit the data\/observations as best as possible. Visually, this will mean that the line is drawn to pass as close as possible<\/em> to all the observations.<\/p>\n

 <\/p>\n

In fact, linear regression is also called OLS regression<\/em>, which stands for ordinary least squares<\/em>.<\/strong> The least squares<\/em>\u00a0concept comes from the fact that to minimize the distances of the observations to the prediction line, we need to first square them before adding them together[4]<\/sup><\/a> — just like we needed to do that in the calculation of the variance and the sum of squares (or the distances would cancel each other out)[5]<\/sup><\/a>.<\/p>\n

 <\/p>\n

But how do we ensure that the regression line minimizes the residuals? The next section explains.<\/p>\n

  1. This something else<\/em> is an 'unobserved variable', or a variable not included in the model (even though we could speculate about it). This type of unobserved variable\/s is the source for the unexplained variance in y<\/em>. ↵<\/a><\/li>
  2. This line of course does not exist, it is a heuristic device. ↵<\/a><\/li>
  3. This is the small-case Greek letter e<\/em>, \u03b5<\/em> [EHpsilon]. ↵<\/a><\/li>
  4. I.e., \"\Sigma{(y-\hat{y})^2}\". ↵<\/a><\/li>
  5. The ordinary<\/em> part is there to differentiate between another regression version called generalized least squares regression<\/em>, or GLS<\/em> regression (not discussed here). ↵<\/a><\/li><\/ol><\/div>","protected":false},"author":533,"menu_order":3,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"class_list":["post-135","chapter","type-chapter","status-publish","hentry"],"part":128,"_links":{"self":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters\/135","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/users\/533"}],"version-history":[{"count":8,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters\/135\/revisions"}],"predecessor-version":[{"id":2147,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters\/135\/revisions\/2147"}],"part":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/parts\/128"}],"metadata":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapters\/135\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/media?parent=135"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/pressbooks\/v2\/chapter-type?post=135"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/contributor?post=135"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/simplestats\/wp-json\/wp\/v2\/license?post=135"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}