Machine Learningmachine learning

Which Machine Learning Algorithm Should You Pick? A Decision-Tree Walk-Through

P&C

Prompt & Circuit Editorial

May 14, 2026 · 12 Min Read

Which Machine Learning Algorithm Should You Pick? A Decision-Tree Walk-Through

TL;DR

  • Need to predict a known outcome? That is supervised learning. Have no labels and want to find structure? That is unsupervised learning.
  • Predicting a number: start with linear regression.
  • Predicting a category: start with logistic regression, then try random forests or gradient boosting.
  • Grouping similar records: start with k-means, and switch to DBSCAN for irregular cluster shapes.
  • Reducing many features to a few: start with PCA. Use t-SNE for 2D or 3D visualisation.
  • Do exploratory data analysis first. Sometimes a query or a pivot table answers the question without a model.

The question every beginner asks

"Which machine learning algorithm should I use?" There is no universally best answer. The right model depends on:

  • what you are trying to find out
  • what kind of data you have, and whether it has labels
  • how clean the data is
  • whether you need to explain the predictions
  • how much compute and training time you can afford

A fraud detector and a house-price estimator both use machine learning, but they are different problems. So start with questions, not algorithms.

The decision tree

The first question splits machine learning into its two biggest families: are you trying to predict something?

Click to expand and zoom

That first split shapes everything after it: whether you need labelled data, which metrics you use and how you validate the model.

Supervised learning: predicting outcomes

If you have historical examples with the answer attached (a label or target column), you are in supervised learning. The next question is whether that target is a number or a category.

Regression: predicting numbers

Use regression when the target is continuous: prices, demand, temperature, revenue.

Start with linear regression. It is fast, its coefficients are easy to interpret and it gives you a baseline that any more complex model has to beat. If the features are highly correlated or numerous, regularised variants such as ridge or lasso are the natural next step.

Classification: predicting categories

Use classification when the target is a label: spam or not spam, fraud or legitimate, churned or retained.

Start with logistic regression. Despite the name, it is a classification algorithm. It trains quickly, produces class probabilities and is easy to explain, which makes it a strong baseline.

Random forests and gradient boosting

The usual next step for tabular data is a tree ensemble. The scikit-learn documentation describes the trade-off this way: random forests build deep trees and average their votes, which can need many trees and a lot of computation, while gradient boosting builds shallow trees in sequence that are faster to fit and predict. Its advice is to try both and compare them on your own problem.

Two practical notes from the same documentation:

  • HistGradientBoostingClassifier can be orders of magnitude faster than GradientBoostingClassifier once you have tens of thousands of samples.
  • The histogram-based version supports missing values and categorical features natively, so it often needs less preprocessing than one-hot encoding everything.

Is deep learning necessary?

For most tabular problems, no. A 2022 benchmark by Grinsztajn, Oyallon and Varoquaux across 45 tabular datasets found that tree-based models such as XGBoost and random forests remained state of the art on medium-sized data (around 10,000 samples), even before counting their speed advantage. Deep learning shines on images, audio and text; on spreadsheets, start with trees.

Unsupervised learning: finding structure

If you have no labels, you are exploring rather than predicting. The goal is usually one of two things: grouping records, or compressing many features into a few.

Clustering

Clustering groups similar records together, for example for customer segmentation or exploratory analysis.

Start with k-means. It is fast and intuitive, but it has two catches. You must specify the number of clusters, and the scikit-learn documentation notes that it assumes clusters are convex and isotropic, so it "responds poorly to elongated clusters, or manifolds with irregular shapes."

DBSCAN is the common alternative. It finds clusters of any shape based on density, labels outlying points as noise and does not ask for a cluster count. You tune a neighbourhood size (eps) and a minimum number of samples instead.

Dimensionality reduction: PCA vs t-SNE

Both reduce the number of features, but they serve different purposes.

  • PCA is fast, deterministic and useful both for visualisation and as a preprocessing step before another model.
  • t-SNE is mainly for visualisation. The scikit-learn documentation notes that it is computationally expensive (hours on million-sample datasets where PCA takes seconds or minutes), that different random seeds can give different embeddings and that global structure is not explicitly preserved.

Use PCA in pipelines, and t-SNE to look at your data.

Before any model: exploratory data analysis

Exploratory data analysis (EDA) is the step beginners skip most often. Before modelling, check:

  • Are values missing?
  • Are features strongly correlated?
  • Are outliers dominating?
  • Is there target leakage, a feature that would not be available at prediction time?
  • Are the classes imbalanced?

Some questions are not machine learning problems at all. "Which products did our highest-spending customers buy?" or "How has the cancellation rate changed this year?" are answered faster with a GROUP BY in SQL or pandas than with any model.

A practical workflow

Click to expand and zoom
  1. Start simple. Fit linear or logistic regression, k-means or PCA first. Without a baseline you cannot tell whether a complex model is worth its cost.
  2. Validate properly. Use cross-validation and a metric that fits the task: MAE or RMSE for regression, ROC AUC, precision and recall for classification (especially with imbalanced classes), silhouette score for clustering.
  3. Then iterate. Try tree ensembles, tune hyperparameters and engineer features, keeping only changes that beat the baseline.
  4. Interpret. Feature importances and coefficients often matter as much as the score, because they tell you what is driving the outcome.

Runnable examples

The following script uses scikit-learn's built-in datasets, so it runs as-is. It fits a regression baseline, compares a classification baseline with two tree ensembles using the same cross-validation, and runs k-means and PCA on scaled data.

from sklearn.cluster import KMeans
from sklearn.datasets import load_breast_cancer, load_diabetes, load_iris
from sklearn.decomposition import PCA
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import mean_absolute_error, r2_score, silhouette_score
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Regression baseline
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
reg = LinearRegression().fit(X_train, y_train)
pred = reg.predict(X_test)
print(f"MAE: {mean_absolute_error(y_test, pred):.1f}  R^2: {r2_score(y_test, pred):.3f}")

# Classification: baseline vs tree ensembles, same 5-fold CV
X, y = load_breast_cancer(return_X_y=True)
models = {
    "logistic regression": make_pipeline(StandardScaler(), LogisticRegression()),
    "random forest": RandomForestClassifier(random_state=42),
    "hist gradient boosting": HistGradientBoostingClassifier(random_state=42),
}
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
    print(f"{name}: ROC AUC {scores.mean():.3f} (+/- {scores.std():.3f})")

# Unsupervised: scale, cluster, reduce
X, _ = load_iris(return_X_y=True)
X_scaled = StandardScaler().fit_transform(X)
kmeans = KMeans(n_clusters=3, random_state=42).fit(X_scaled)
print(f"Silhouette: {silhouette_score(X_scaled, kmeans.labels_):.3f}")
pca = PCA(n_components=2).fit(X_scaled)
print("Variance explained by 2 components:", pca.explained_variance_ratio_.round(3))

With scikit-learn 1.9.1, our run printed:

MAE: 42.8  R^2: 0.453
logistic regression: ROC AUC 0.995 (+/- 0.004)
random forest: ROC AUC 0.991 (+/- 0.007)
hist gradient boosting: ROC AUC 0.993 (+/- 0.004)
Silhouette: 0.480
Variance explained by 2 components: [0.73  0.229]

On this small, clean dataset the scaled logistic regression baseline is as good as either ensemble, which is exactly why you fit the baseline first. Scaling matters here too: logistic regression, k-means and PCA are all sensitive to feature scale, while tree models are not.

FAQ

Which algorithms should beginners learn first? Linear regression, logistic regression, decision trees, k-means and PCA. Together they cover the core ideas behind most other methods.

Random forest or gradient boosting? Try both. Neither wins on every dataset, as the example above shows, and both should be compared against a simple baseline using the same cross-validation.

Accuracy or interpretability? It depends on the stakes. In regulated settings such as lending or healthcare, being able to explain a prediction can matter more than a small gain in accuracy.

Final thoughts

You do not need to memorise every algorithm. The skill is diagnosing the problem: is there a target, what type is it, and what does the data look like? Answer those questions, fit a simple baseline, and the right model usually becomes clear from there.

Sources

Tags:machine learningscikit-learnsupervised learningunsupervised learningmodel selection
Share Article

Responses (0)

Sign in to join the conversation and share your thoughts.

P&C

Prompt & Circuit Editorial

Contributing Editor

Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.