4. The Simple Linear Regression Model
Deriving OLS and interpreting the slope, intercept, predictions and residuals
Population model and least-squares criterion
The simple linear regression model writes the outcome as a linear function of one explanatory variable and an unobserved disturbance:
The intercept β0 is the conditional mean at X=0 if E[u|X]=0 and X=0 lies in the meaningful range of the data. The slope β1 measures a one-unit change in the conditional mean of Y for a one-unit change in X, conditional on the model. The error term gathers unobserved determinants; it is not simply a synonym for mistakes by the researcher, although mismeasurement can be one of its components.
The ordinary least squares estimator selects b0 and b1 to minimise squared fitted residuals:
Squaring makes negative and positive residuals contribute positively and gives greater weight to larger absolute errors. This criterion is a well-defined sample operation, independent of whether the regressor was randomly assigned. Its causal interpretation is a separate question.
Deriving the slope and intercept
Differentiate the sum of squared residuals with respect to b0 and b1 and set the derivatives equal to zero. The first normal equation states that the sample residuals sum to zero; it implies that the fitted line passes through the sample averages (X̄,Ȳ). Substituting b0=Ȳ−b1X̄ into the second normal equation yields
Thus the slope is the sample covariance divided by the sample variance of X, using the same normalisation in numerator and denominator; equivalently, it is the centred cross-product sum divided by the centred squared-X sum. A flat X with no variation makes the denominator zero: the slope cannot be estimated. The numerator is positive when high-X observations systematically coincide with high-Y observations and negative in the opposite case.
Residuals, fitted values and OLS identities
The fitted value is Ŷi=β̂0+β̂1Xi and the residual is ûi=Yi−Ŷi. With an intercept, OLS normal equations guarantee Σûi=0 and ΣXiûi=0. Consequently the sample covariance between fitted values and residuals is zero, and mean fitted Y equals observed mean Y. These are algebraic properties of the fitted sample, not proof that the unobserved population disturbance is independent of X.
The exact sample variance decomposition is
The three terms are respectively total sum of squares (TSS), explained sum of squares (ESS) and residual sum of squares (RSS, also called SSR in some conventions). It follows that 0≤R2≤1 in an OLS regression containing an intercept, assuming a nonconstant outcome.
Binary regressors: comparing means by regression
Suppose D takes values zero and one and Yi=β0+β1Di+ui. OLS sets β̂0 equal to the observed mean outcome among D=0 and β̂0+β̂1 equal to the mean among D=1, so
A regression with one treatment dummy and an intercept exactly reproduces the difference in sample means. If the treatment was randomly assigned and there is no problematic attrition or interference, it can estimate an average causal treatment effect. If the dummy indicates self-selected college attendance, the coefficient may reflect pre-existing differences in ability or family background.
Rescaling and functional forms
A coefficient depends on units. If X is measured in thousands instead of individual euros, its coefficient increases by a factor of one thousand, while the fitted values and substantive implications remain unchanged. Changing the scale of Y multiplies the coefficient and fitted residuals by the same factor. Transformations must always be interpreted in terms of original economic units.
A level-level model Y=β0+β1X+u has β1 units of Y per unit of X. A log-level model log(Y)=β0+β1X+u has an approximate 100β1-percent change in Y per unit increase in X; the exact proportional effect for a unit change is exp(β1)−1 under appropriate log-scale interpretation. In level-log models the outcome changes by approximately β1/100 units for a 1% change in X. In log-log models β1 is the elasticity of Y with respect to X.
Why an estimated line is not the data-generating process
Regression estimates a best-fitting sample relationship. Even if the population conditional mean is nonlinear, the population least-squares line remains meaningful as the best linear prediction under squared-error loss. If Y is driven by X2, a fitted slope may vary markedly when the observed range of X changes, despite technically correct OLS computation. The fitted line is an approximation; the economic researcher must argue when it is a useful approximation and whether it has an acceptable causal meaning.
The population linear projection
To understand OLS without assuming a truly linear economic mechanism, define the population linear projection as the choice of α and β minimising E[(Y−α−βX)2]. Its slope is β=Cov(X,Y)/Var(X) whenever these moments exist and Var(X)>0. The projection error e=Y−α−βX is orthogonal to the constant and X: E[e]=0 and E[Xe]=0. However E[e|X]=0 need not hold if the true conditional expectation is nonlinear. The difference matters: sample orthogonality is an algebraic result, while conditional mean independence is a richer property of the population distribution.
For a nonlinear conditional expectation, the linear projection is a weighted compromise across observed values of X. The slope can be a useful summary of average linear association and can provide a good forecasting benchmark over the sample's covariate range, even when it should not be interpreted as an invariant structural derivative. The researcher must identify what the coefficient means before attaching a policy interpretation.
Leverage, outliers and influential observations
An observation with an unusual regressor value can have high leverage, meaning it has substantial potential to influence the fitted line. In matrix OLS, leverage hi is the corresponding diagonal element of Px=X(X′X)-1X′. Large residuals and high leverage are distinct properties: an observation can be far from the predictor centre but lie almost exactly on the fitted line; another can have an ordinary X value but an unexpectedly high or low Y.
Influence measures combine leverage and residual size to assess the effect of removing observations. A suspicious record might indicate coding or measurement error, but genuine rare observations may carry essential economic information. Deletion based solely on whether a point weakens statistical significance undermines empirical credibility. Report meaningful changes under sensitivity checks and justify exclusions using defensible data-quality rules.