Machine Learninglinear-regression

Linear Regression in Practice: Fitting, Evaluating and Checking Assumptions in Python

P&C

Prompt & Circuit Editorial

May 25, 2026 · 12 Min Read

Linear Regression in Practice: Fitting, Evaluating and Checking Assumptions in Python

TL;DR

  • Linear regression predicts a continuous target as a weighted sum of features plus an intercept.
  • It is fitted by minimising the sum of squared residuals, either iteratively (gradient descent) or directly (a least-squares solver).
  • Judge it with R², RMSE and MAE on held-out data, and with residual diagnostics.
  • Its inference relies on assumptions: linearity, independent errors, constant error variance, roughly normal residuals and no severe multicollinearity.
  • Use it as a baseline and wherever interpretable coefficients matter.

What linear regression does

Linear regression fits a line (one feature) or a hyperplane (several features) through the data. Each coefficient says how much the prediction changes when its feature increases by one unit while the others are held constant. That interpretability is why it remains the first model to try: if a linear model already explains most of the variance, a more complex model has to earn its extra cost, and if it does not, you still have a baseline to compare against.

Variants

VariantPredictorsTargetsModel
Simple11y=β0+β1x+εy = \beta_0 + \beta_1 x + \varepsilon
Multiplep1y=β0+β1x1+⋯+βpxp+εy = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p + \varepsilon
Multivariate (multi-output)pmY=XB+E\mathbf{Y} = \mathbf{X}\mathbf{B} + \mathbf{E}

scikit-learn's LinearRegression handles the multi-output case directly: fit accepts y with shape (n_samples, n_targets), and coef_ then has shape (n_targets, n_features).

The objective: least squares

Ordinary least squares chooses the coefficients that minimise the residual sum of squares:

RSS=∑i=1n(yi−y^i)2\text{RSS} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2

Dividing by n gives the mean squared error (MSE), which has the same minimiser and is the usual loss for gradient descent:

MSE=1n∑i=1n(yi−y^i)2\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2

Solving it: gradient descent

With a design matrix XX that includes a column of ones for the intercept, the gradient of the MSE is

∇βMSE=2nX⊤(Xβ−y)\nabla_\beta \text{MSE} = \frac{2}{n} X^\top (X\beta - y)

and each step updates β←β−α∇βMSE\beta \leftarrow \beta - \alpha \nabla_\beta \text{MSE} for a learning rate α\alpha. Features on very different scales make the loss surface badly conditioned, so standardise them first and convert the coefficients back afterwards.

import numpy as np

rng = np.random.default_rng(42)
n = 500
size = rng.uniform(50, 250, n)      # floor area, square metres
age = rng.uniform(0, 60, n)         # building age, years
y = 40_000 + 1_800 * size - 900 * age + rng.normal(0, 20_000, n)

X = np.column_stack([size, age])
X_design = np.column_stack([np.ones(n), X])   # intercept column


def gradient_descent(X, y, lr=0.1, epochs=5_000):
    """Batch gradient descent on MSE. X must include an intercept column."""
    n_samples, n_features = X.shape
    beta = np.zeros(n_features)
    for _ in range(epochs):
        error = X @ beta - y
        grad = (2 / n_samples) * X.T @ error
        beta -= lr * grad
    return beta


mean, std = X.mean(axis=0), X.std(axis=0)
X_scaled = np.column_stack([np.ones(n), (X - mean) / std])
beta_scaled = gradient_descent(X_scaled, y)

# Convert back to original units
slopes = beta_scaled[1:] / std
intercept = beta_scaled[0] - np.sum(beta_scaled[1:] * mean / std)
print(intercept, slopes)

Gradient descent, and especially its mini-batch and stochastic forms, is useful when data is too large to process in one pass or arrives as a stream.

Solving it: the normal equation and least-squares solvers

Setting the gradient to zero gives the normal equation:

β=(X⊤X)−1X⊤y\beta = (X^\top X)^{-1} X^\top y

Avoid computing the inverse explicitly. It is numerically fragile when features are correlated, and it fails outright when X⊤XX^\top X is singular. Use a least-squares solver instead:

beta, residuals, rank, singular_values = np.linalg.lstsq(X_design, y, rcond=None)
print(beta)   # intercept, size slope, age slope

np.linalg.lstsq returns the solution together with the rank of X, so you can detect rank deficiency; when there are many minimising solutions it returns the one with the smallest norm. On the synthetic data above, gradient descent and lstsq agree to many decimal places. scikit-learn's LinearRegression also uses a least-squares solver (scipy.linalg.lstsq) for dense data.

Gradient descent versus a direct solver

AspectGradient descentDirect least-squares solver
IterationsManyNone (one factorisation)
TuningLearning rate, epochs, feature scalingNone
CostScales linearly with rows per epoch; supports mini-batchesGrows roughly with the cube of the number of features
Streaming dataNatural fitNeeds all data at once
Collinear featuresConverges slowlySVD-based solvers handle rank deficiency

For typical tabular problems with up to thousands of features, a direct solver is simpler and exact. Gradient descent earns its place with very large or streaming data, or as a teaching tool for the optimisers used in deep learning.

Fitting and evaluating with scikit-learn

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)

print("coefficients:", model.intercept_, model.coef_)
print("R2:  ", r2_score(y_test, pred))
print("RMSE:", np.sqrt(mean_squared_error(y_test, pred)))
print("MAE: ", mean_absolute_error(y_test, pred))
MetricMeaningRange
R²Share of variance explained relative to predicting the meanAt most 1. A constant mean predictor scores 0, and a worse model can score below 0
RMSETypical error in the target's units, weighting large errors heavily0 and up
MAEAverage absolute error in the target's units, less sensitive to outliers than RMSE0 and up

R2=1−RSSTSS,TSS=∑i(yi−yˉ)2R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}, \quad \text{TSS} = \sum_i (y_i - \bar{y})^2

Always compute these on data the model did not see. A high R² on training data says little about generalisation, and R² rises whenever you add features, so use adjusted R² or cross-validation when comparing models with different numbers of predictors.

Assumptions and how to check them

AssumptionWhy it mattersCheckRemedy
LinearityThe model only captures linear effectsPlot residuals against fitted values and each featureTransform features, add polynomial or interaction terms
Independent errorsCorrelated errors make standard errors too smallDurbin-Watson statistic; residuals over timeAdd lagged features, use time-series models
Constant variance (homoscedasticity)Unequal variance distorts standard errors and intervalsResiduals vs fitted values (look for a funnel)Transform the target, weighted least squares, robust standard errors
Normal residualsNeeded for exact small-sample p-values and intervalsQ-Q plotTransform the target, bootstrap intervals
Limited multicollinearityInflates coefficient variance and makes them unstableVariance inflation factor (VIF)Drop or combine features, use ridge or lasso

statsmodels provides both numeric checks:

from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.stats.stattools import durbin_watson

vifs = [variance_inflation_factor(X_design, i) for i in range(1, X_design.shape[1])]
print("VIF:", vifs)
print("Durbin-Watson:", durbin_watson(y_train - model.predict(X_train)))

The statsmodels documentation treats a VIF above 5 as a sign of high collinearity. The Durbin-Watson statistic always lies between 0 and 4: values near 2 suggest no serial correlation, values towards 0 suggest positive correlation and values towards 4 negative correlation. On the independent synthetic data above, both VIFs are close to 1 and the statistic is close to 2.

Violations do not always invalidate predictions. They matter most when you rely on coefficients, p-values or confidence intervals.

Where linear regression is used

  • Pricing and demand: how price, promotion and season relate to units sold.
  • Real estate: price as a function of floor area, age and location features.
  • Finance: factor models that relate an asset's returns to market factors.
  • Health research: relating a continuous measurement such as blood pressure to lifestyle variables.
  • Agriculture: yield as a function of rainfall and fertiliser.

In each case the attraction is the same: a coefficient that reads as "one more unit of X is associated with this much more Y, other things equal". Associated, not caused; observational data alone does not establish causation.

Checklist

  1. Clean the data: handle missing values and encode categorical features.
  2. Explore: scatter plots and a correlation matrix.
  3. Split into training and test sets before fitting anything.
  4. Fit with LinearRegression (or np.linalg.lstsq); reach for gradient descent only for very large or streaming data.
  5. Evaluate R², RMSE and MAE on the test set.
  6. Check residual plots, VIF and, for ordered data, Durbin-Watson.
  7. Iterate: transform features, add interactions, or add ridge or lasso regularisation.
  8. Persist the fitted model and serve it behind an API when it is ready.

FAQ

When should I avoid linear regression? When the relationship is clearly non-linear and simple transformations do not fix it, or when interactions are complex and unknown. Tree-based models or generalised additive models are common next steps, and the linear model remains the baseline to beat.

Does a high R² mean the model is not overfitting? No. Training R² increases with every added predictor. Use adjusted R², a held-out test set or cross-validation.

How do I choose a learning rate for gradient descent? Standardise the features first, then try a range such as 0.001 to 0.1. If the loss oscillates or grows, lower the rate; if it falls very slowly, raise it.

Can linear regression be used for classification? It is not designed for it: predictions are unbounded and not probabilities. Use logistic regression or another classifier for discrete labels.

Sources

Tags:linear-regressionscikit-learngradient-descentmodel-evaluationpython
Share Article

Responses (0)

Sign in to join the conversation and share your thoughts.

P&C

Prompt & Circuit Editorial

Contributing Editor

Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.