Linear Regression in Practice: Fitting, Evaluating and Checking Assumptions in Python
Prompt & Circuit Editorial
May 25, 2026 · 12 Min Read

TL;DR
- Linear regression predicts a continuous target as a weighted sum of features plus an intercept.
- It is fitted by minimising the sum of squared residuals, either iteratively (gradient descent) or directly (a least-squares solver).
- Judge it with R², RMSE and MAE on held-out data, and with residual diagnostics.
- Its inference relies on assumptions: linearity, independent errors, constant error variance, roughly normal residuals and no severe multicollinearity.
- Use it as a baseline and wherever interpretable coefficients matter.
What linear regression does
Linear regression fits a line (one feature) or a hyperplane (several features) through the data. Each coefficient says how much the prediction changes when its feature increases by one unit while the others are held constant. That interpretability is why it remains the first model to try: if a linear model already explains most of the variance, a more complex model has to earn its extra cost, and if it does not, you still have a baseline to compare against.
Variants
| Variant | Predictors | Targets | Model |
|---|---|---|---|
| Simple | 1 | 1 | |
| Multiple | p | 1 | |
| Multivariate (multi-output) | p | m |
scikit-learn's LinearRegression handles the multi-output case directly: fit accepts y with shape (n_samples, n_targets), and coef_ then has shape (n_targets, n_features).
The objective: least squares
Ordinary least squares chooses the coefficients that minimise the residual sum of squares:
Dividing by n gives the mean squared error (MSE), which has the same minimiser and is the usual loss for gradient descent:
Solving it: gradient descent
With a design matrix that includes a column of ones for the intercept, the gradient of the MSE is
and each step updates for a learning rate . Features on very different scales make the loss surface badly conditioned, so standardise them first and convert the coefficients back afterwards.
import numpy as np
rng = np.random.default_rng(42)
n = 500
size = rng.uniform(50, 250, n) # floor area, square metres
age = rng.uniform(0, 60, n) # building age, years
y = 40_000 + 1_800 * size - 900 * age + rng.normal(0, 20_000, n)
X = np.column_stack([size, age])
X_design = np.column_stack([np.ones(n), X]) # intercept column
def gradient_descent(X, y, lr=0.1, epochs=5_000):
"""Batch gradient descent on MSE. X must include an intercept column."""
n_samples, n_features = X.shape
beta = np.zeros(n_features)
for _ in range(epochs):
error = X @ beta - y
grad = (2 / n_samples) * X.T @ error
beta -= lr * grad
return beta
mean, std = X.mean(axis=0), X.std(axis=0)
X_scaled = np.column_stack([np.ones(n), (X - mean) / std])
beta_scaled = gradient_descent(X_scaled, y)
# Convert back to original units
slopes = beta_scaled[1:] / std
intercept = beta_scaled[0] - np.sum(beta_scaled[1:] * mean / std)
print(intercept, slopes)
Gradient descent, and especially its mini-batch and stochastic forms, is useful when data is too large to process in one pass or arrives as a stream.
Solving it: the normal equation and least-squares solvers
Setting the gradient to zero gives the normal equation:
Avoid computing the inverse explicitly. It is numerically fragile when features are correlated, and it fails outright when is singular. Use a least-squares solver instead:
beta, residuals, rank, singular_values = np.linalg.lstsq(X_design, y, rcond=None)
print(beta) # intercept, size slope, age slope
np.linalg.lstsq returns the solution together with the rank of X, so you can detect rank deficiency; when there are many minimising solutions it returns the one with the smallest norm. On the synthetic data above, gradient descent and lstsq agree to many decimal places. scikit-learn's LinearRegression also uses a least-squares solver (scipy.linalg.lstsq) for dense data.
Gradient descent versus a direct solver
| Aspect | Gradient descent | Direct least-squares solver |
|---|---|---|
| Iterations | Many | None (one factorisation) |
| Tuning | Learning rate, epochs, feature scaling | None |
| Cost | Scales linearly with rows per epoch; supports mini-batches | Grows roughly with the cube of the number of features |
| Streaming data | Natural fit | Needs all data at once |
| Collinear features | Converges slowly | SVD-based solvers handle rank deficiency |
For typical tabular problems with up to thousands of features, a direct solver is simpler and exact. Gradient descent earns its place with very large or streaming data, or as a teaching tool for the optimisers used in deep learning.
Fitting and evaluating with scikit-learn
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)
print("coefficients:", model.intercept_, model.coef_)
print("R2: ", r2_score(y_test, pred))
print("RMSE:", np.sqrt(mean_squared_error(y_test, pred)))
print("MAE: ", mean_absolute_error(y_test, pred))
| Metric | Meaning | Range |
|---|---|---|
| R² | Share of variance explained relative to predicting the mean | At most 1. A constant mean predictor scores 0, and a worse model can score below 0 |
| RMSE | Typical error in the target's units, weighting large errors heavily | 0 and up |
| MAE | Average absolute error in the target's units, less sensitive to outliers than RMSE | 0 and up |
Always compute these on data the model did not see. A high R² on training data says little about generalisation, and R² rises whenever you add features, so use adjusted R² or cross-validation when comparing models with different numbers of predictors.
Assumptions and how to check them
| Assumption | Why it matters | Check | Remedy |
|---|---|---|---|
| Linearity | The model only captures linear effects | Plot residuals against fitted values and each feature | Transform features, add polynomial or interaction terms |
| Independent errors | Correlated errors make standard errors too small | Durbin-Watson statistic; residuals over time | Add lagged features, use time-series models |
| Constant variance (homoscedasticity) | Unequal variance distorts standard errors and intervals | Residuals vs fitted values (look for a funnel) | Transform the target, weighted least squares, robust standard errors |
| Normal residuals | Needed for exact small-sample p-values and intervals | Q-Q plot | Transform the target, bootstrap intervals |
| Limited multicollinearity | Inflates coefficient variance and makes them unstable | Variance inflation factor (VIF) | Drop or combine features, use ridge or lasso |
statsmodels provides both numeric checks:
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.stats.stattools import durbin_watson
vifs = [variance_inflation_factor(X_design, i) for i in range(1, X_design.shape[1])]
print("VIF:", vifs)
print("Durbin-Watson:", durbin_watson(y_train - model.predict(X_train)))
The statsmodels documentation treats a VIF above 5 as a sign of high collinearity. The Durbin-Watson statistic always lies between 0 and 4: values near 2 suggest no serial correlation, values towards 0 suggest positive correlation and values towards 4 negative correlation. On the independent synthetic data above, both VIFs are close to 1 and the statistic is close to 2.
Violations do not always invalidate predictions. They matter most when you rely on coefficients, p-values or confidence intervals.
Where linear regression is used
- Pricing and demand: how price, promotion and season relate to units sold.
- Real estate: price as a function of floor area, age and location features.
- Finance: factor models that relate an asset's returns to market factors.
- Health research: relating a continuous measurement such as blood pressure to lifestyle variables.
- Agriculture: yield as a function of rainfall and fertiliser.
In each case the attraction is the same: a coefficient that reads as "one more unit of X is associated with this much more Y, other things equal". Associated, not caused; observational data alone does not establish causation.
Checklist
- Clean the data: handle missing values and encode categorical features.
- Explore: scatter plots and a correlation matrix.
- Split into training and test sets before fitting anything.
- Fit with
LinearRegression(ornp.linalg.lstsq); reach for gradient descent only for very large or streaming data. - Evaluate R², RMSE and MAE on the test set.
- Check residual plots, VIF and, for ordered data, Durbin-Watson.
- Iterate: transform features, add interactions, or add ridge or lasso regularisation.
- Persist the fitted model and serve it behind an API when it is ready.
FAQ
When should I avoid linear regression? When the relationship is clearly non-linear and simple transformations do not fix it, or when interactions are complex and unknown. Tree-based models or generalised additive models are common next steps, and the linear model remains the baseline to beat.
Does a high R² mean the model is not overfitting? No. Training R² increases with every added predictor. Use adjusted R², a held-out test set or cross-validation.
How do I choose a learning rate for gradient descent? Standardise the features first, then try a range such as 0.001 to 0.1. If the loss oscillates or grows, lower the rate; if it falls very slowly, raise it.
Can linear regression be used for classification? It is not designed for it: predictions are unbounded and not probabilities. Use logistic regression or another classifier for discrete labels.
Sources
Responses (0)
Sign in to join the conversation and share your thoughts.

Prompt & Circuit Editorial
Contributing Editor
Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.
Continue Reading

Implementing Hybrid Semantic and Lexical Search for RAG with BM25 and Reciprocal Rank Fusion
Dense embeddings miss exact identifiers and BM25 misses paraphrases. This walk-through combines both in Python with rank_bm25, sentence-transformers and Reciprocal Rank Fusion, and explains the trade-offs before you move it to a search engine.

Demystifying LangChain Models: From LLMs to Embeddings in Plain English
A plain-English guide to LangChain's model layer: chat models, legacy text-completion LLMs, embedding models, invoke(), key parameters, and how embeddings power semantic search, with code for the current LangChain 1.x APIs.

Building Human-in-the-Loop Agentic Workflows with LangGraph
How to pause a LangGraph workflow for human approval with interrupt(), resume it with Command, and design review steps that hold up in production.