8. Analysis of Variance and Model Fit
ANOVA, R-squared, residual diagnostics and predictive accuracy
Orthogonality and decomposition of sums of squares
OLS separates observed outcomes into fitted values and residuals: Yi=Ŷi+ûi. With an intercept, sample orthogonality implies that the total sum of squared deviations from Ȳ is the sum of the squared fitted deviations and squared residuals:
Dividing by TSS gives R2=ESS/TSS=1−RSS/TSS. A value near one means the linear predictors explain much of the sample variation in Y around its sample mean. It does not demonstrate an unbiased causal effect; time trends, common shocks and mechanically related variables can generate very high R2 values without a credible structural interpretation.
The meaning of adjusted R-squared
Adding a regressor cannot increase the in-sample RSS when OLS is recomputed, so ordinary R2 never decreases in nested models. Adjusted R2 penalises the number of estimated parameters:
The adjustment can fall when the added variable improves fit only slightly. It is not an unbiased estimate of out-of-sample prediction quality, and it is not a causal variable-selection rule. Comparisons between models using different dependent-variable transformations require particular caution: R2 values for Y and ln(Y) are not directly comparable measures of fit on the same outcome scale.
Standard error of regression and forecast errors
The residual standard error is SER=√(RSS/(n−k−1)). It estimates the scale of errors in units of Y under relevant assumptions. A predictive application cares about error on data not used to estimate the model. If the fitted model is evaluated on an independent validation sample, a common performance metric is mean squared prediction error:
Root MSPE has the units of the predicted variable and is more interpretable than raw squared errors. For forecasting, validation observations must lie later in time than training observations; randomly shuffling a time series can leak future information into the model-building process.
Residual plots and specification checks
Residual plots against fitted values, each regressor, time and group indicators can reveal outliers, heteroskedasticity, nonlinearity and serial patterns. A residual scatter that fans out suggests a variance model that depends on X. A curved residual pattern suggests omitted nonlinear terms. Serial runs of positive and negative residuals in time-series data suggest dynamic misspecification or correlated disturbances.
However, OLS residuals are orthogonal to included regressors by construction. A zero sample residual–regressor covariance therefore does not validate the zero-conditional-mean assumption. Specification tests are useful diagnostics but cannot by themselves identify unobserved counterfactuals or prove exogeneity.
Overfitting and out-of-sample validation
When the number of regressors becomes large relative to the number of observations, in-sample fit can improve even if the model loses predictive reliability. With n parameters and n observations, it may be possible to interpolate the dataset perfectly, leaving no honest out-of-sample guarantee. Cross-validation, independent test sets and regularisation target predictive performance rather than merely the fitted-data RSS.
| Statistic | What it measures | What it does not prove |
|---|---|---|
| R2 | In-sample variation explained by projection | Causal identification |
| Adjusted R2 | Fit penalised for model size | Best prediction on new data |
| SER | Residual scale in outcome units | No specification bias |
| Out-of-sample RMSE | Typical forecast error in outcome units | Structural effect |
| t-statistic | Signal relative to a specified SE | Economic importance |
Variance in regression output
Statistical software often reports an analysis-of-variance table with model sum of squares, residual sum of squares, their degrees of freedom and the corresponding mean squares. When all conventional homoskedastic assumptions hold, the ratio of model and residual mean squares can provide an overall F-test. With heteroskedasticity, a robust Wald test should replace the conventional sum-of-squares comparison for general joint hypotheses.
Sample fit, causal validity and spurious precision
Three models can report identical R2 values while offering very different information. A randomised treatment regression with a low R2 can identify a credible policy effect, because treatment may explain only a small fraction of individual outcome variation. A high-dimensional prediction model with high cross-validated accuracy may provide excellent forecasts without identifying any causal intervention. A trending time-series regression between unrelated integrated variables may have extremely high R2 while standard t-based inference is invalid. Evaluating these models with one fit statistic is a category error.
Decomposing explanatory contributions
In multiple regression, explanatory contributions of correlated regressors cannot always be separated uniquely. If schooling and parental income are correlated, attributing explained wage variation to each variable depends on the decomposition procedure or order of inclusion. A partial R2 for a selected regressor group measures its incremental fit conditional on the remaining regressors; it should not be interpreted as a unique causal share of the outcome.
Partial R2 can be defined through restricted and unrestricted residual sums of squares: R2partial=(RSSrestricted−RSSunrestricted)/RSSrestricted. The numerator measures the fit improvement from adding the selected regressors, while the denominator is the unexplained variation in the restricted model. It can be useful for assessing a conditional relationship or a first stage, but does not validate the exclusion restriction of an instrument.
Confidence intervals for a mean and prediction intervals for a person
An interval for the conditional mean answers how precisely average Y at x0 has been estimated. A prediction interval asks where a new individual outcome Y0 may fall. The second interval includes both uncertainty about the estimated mean and the new observation’s irreducible disturbance, so it is wider under the same model. Here x0 includes the intercept, s2 is the conventional residual variance estimate, p is the total number of fitted coefficients and h0=x0′(X′X)-1x0.
Under the correct homoskedastic normal linear model and an independent new observation, the two intervals use the formulas below. The extra 1 inside the prediction square root is the new disturbance variance in units of s2. Even with a huge estimation sample, this uncertainty does not disappear. Leverage makes both intervals wider away from well-supported covariate values. R distinguishes interval="confidence" from interval="prediction" in predict.lm. Robust coefficient standard errors alone do not construct a valid heteroskedastic prediction interval: the conditional distribution of the new outcome also needs attention.