Which Machine Learning Algorithm Should You Pick? A Decision-Tree Walk-Through
Prompt & Circuit Editorial
May 14, 2026 · 12 Min Read

TL;DR
- Need to predict a known outcome? That is supervised learning. Have no labels and want to find structure? That is unsupervised learning.
- Predicting a number: start with linear regression.
- Predicting a category: start with logistic regression, then try random forests or gradient boosting.
- Grouping similar records: start with k-means, and switch to DBSCAN for irregular cluster shapes.
- Reducing many features to a few: start with PCA. Use t-SNE for 2D or 3D visualisation.
- Do exploratory data analysis first. Sometimes a query or a pivot table answers the question without a model.
The question every beginner asks
"Which machine learning algorithm should I use?" There is no universally best answer. The right model depends on:
- what you are trying to find out
- what kind of data you have, and whether it has labels
- how clean the data is
- whether you need to explain the predictions
- how much compute and training time you can afford
A fraud detector and a house-price estimator both use machine learning, but they are different problems. So start with questions, not algorithms.
The decision tree
The first question splits machine learning into its two biggest families: are you trying to predict something?
Click to expand and zoom
That first split shapes everything after it: whether you need labelled data, which metrics you use and how you validate the model.
Supervised learning: predicting outcomes
If you have historical examples with the answer attached (a label or target column), you are in supervised learning. The next question is whether that target is a number or a category.
Regression: predicting numbers
Use regression when the target is continuous: prices, demand, temperature, revenue.
Start with linear regression. It is fast, its coefficients are easy to interpret and it gives you a baseline that any more complex model has to beat. If the features are highly correlated or numerous, regularised variants such as ridge or lasso are the natural next step.
Classification: predicting categories
Use classification when the target is a label: spam or not spam, fraud or legitimate, churned or retained.
Start with logistic regression. Despite the name, it is a classification algorithm. It trains quickly, produces class probabilities and is easy to explain, which makes it a strong baseline.
Random forests and gradient boosting
The usual next step for tabular data is a tree ensemble. The scikit-learn documentation describes the trade-off this way: random forests build deep trees and average their votes, which can need many trees and a lot of computation, while gradient boosting builds shallow trees in sequence that are faster to fit and predict. Its advice is to try both and compare them on your own problem.
Two practical notes from the same documentation:
HistGradientBoostingClassifiercan be orders of magnitude faster thanGradientBoostingClassifieronce you have tens of thousands of samples.- The histogram-based version supports missing values and categorical features natively, so it often needs less preprocessing than one-hot encoding everything.
Is deep learning necessary?
For most tabular problems, no. A 2022 benchmark by Grinsztajn, Oyallon and Varoquaux across 45 tabular datasets found that tree-based models such as XGBoost and random forests remained state of the art on medium-sized data (around 10,000 samples), even before counting their speed advantage. Deep learning shines on images, audio and text; on spreadsheets, start with trees.
Unsupervised learning: finding structure
If you have no labels, you are exploring rather than predicting. The goal is usually one of two things: grouping records, or compressing many features into a few.
Clustering
Clustering groups similar records together, for example for customer segmentation or exploratory analysis.
Start with k-means. It is fast and intuitive, but it has two catches. You must specify the number of clusters, and the scikit-learn documentation notes that it assumes clusters are convex and isotropic, so it "responds poorly to elongated clusters, or manifolds with irregular shapes."
DBSCAN is the common alternative. It finds clusters of any shape based on density, labels outlying points as noise and does not ask for a cluster count. You tune a neighbourhood size (eps) and a minimum number of samples instead.
Dimensionality reduction: PCA vs t-SNE
Both reduce the number of features, but they serve different purposes.
- PCA is fast, deterministic and useful both for visualisation and as a preprocessing step before another model.
- t-SNE is mainly for visualisation. The scikit-learn documentation notes that it is computationally expensive (hours on million-sample datasets where PCA takes seconds or minutes), that different random seeds can give different embeddings and that global structure is not explicitly preserved.
Use PCA in pipelines, and t-SNE to look at your data.
Before any model: exploratory data analysis
Exploratory data analysis (EDA) is the step beginners skip most often. Before modelling, check:
- Are values missing?
- Are features strongly correlated?
- Are outliers dominating?
- Is there target leakage, a feature that would not be available at prediction time?
- Are the classes imbalanced?
Some questions are not machine learning problems at all. "Which products did our highest-spending customers buy?" or "How has the cancellation rate changed this year?" are answered faster with a GROUP BY in SQL or pandas than with any model.
A practical workflow
Click to expand and zoom
- Start simple. Fit linear or logistic regression, k-means or PCA first. Without a baseline you cannot tell whether a complex model is worth its cost.
- Validate properly. Use cross-validation and a metric that fits the task: MAE or RMSE for regression, ROC AUC, precision and recall for classification (especially with imbalanced classes), silhouette score for clustering.
- Then iterate. Try tree ensembles, tune hyperparameters and engineer features, keeping only changes that beat the baseline.
- Interpret. Feature importances and coefficients often matter as much as the score, because they tell you what is driving the outcome.
Runnable examples
The following script uses scikit-learn's built-in datasets, so it runs as-is. It fits a regression baseline, compares a classification baseline with two tree ensembles using the same cross-validation, and runs k-means and PCA on scaled data.
from sklearn.cluster import KMeans
from sklearn.datasets import load_breast_cancer, load_diabetes, load_iris
from sklearn.decomposition import PCA
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import mean_absolute_error, r2_score, silhouette_score
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
# Regression baseline
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
reg = LinearRegression().fit(X_train, y_train)
pred = reg.predict(X_test)
print(f"MAE: {mean_absolute_error(y_test, pred):.1f} R^2: {r2_score(y_test, pred):.3f}")
# Classification: baseline vs tree ensembles, same 5-fold CV
X, y = load_breast_cancer(return_X_y=True)
models = {
"logistic regression": make_pipeline(StandardScaler(), LogisticRegression()),
"random forest": RandomForestClassifier(random_state=42),
"hist gradient boosting": HistGradientBoostingClassifier(random_state=42),
}
for name, model in models.items():
scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
print(f"{name}: ROC AUC {scores.mean():.3f} (+/- {scores.std():.3f})")
# Unsupervised: scale, cluster, reduce
X, _ = load_iris(return_X_y=True)
X_scaled = StandardScaler().fit_transform(X)
kmeans = KMeans(n_clusters=3, random_state=42).fit(X_scaled)
print(f"Silhouette: {silhouette_score(X_scaled, kmeans.labels_):.3f}")
pca = PCA(n_components=2).fit(X_scaled)
print("Variance explained by 2 components:", pca.explained_variance_ratio_.round(3))
With scikit-learn 1.9.1, our run printed:
MAE: 42.8 R^2: 0.453
logistic regression: ROC AUC 0.995 (+/- 0.004)
random forest: ROC AUC 0.991 (+/- 0.007)
hist gradient boosting: ROC AUC 0.993 (+/- 0.004)
Silhouette: 0.480
Variance explained by 2 components: [0.73 0.229]
On this small, clean dataset the scaled logistic regression baseline is as good as either ensemble, which is exactly why you fit the baseline first. Scaling matters here too: logistic regression, k-means and PCA are all sensitive to feature scale, while tree models are not.
FAQ
Which algorithms should beginners learn first? Linear regression, logistic regression, decision trees, k-means and PCA. Together they cover the core ideas behind most other methods.
Random forest or gradient boosting? Try both. Neither wins on every dataset, as the example above shows, and both should be compared against a simple baseline using the same cross-validation.
Accuracy or interpretability? It depends on the stakes. In regulated settings such as lending or healthcare, being able to explain a prediction can matter more than a small gain in accuracy.
Final thoughts
You do not need to memorise every algorithm. The skill is diagnosing the problem: is there a target, what type is it, and what does the data look like? Answer those questions, fit a simple baseline, and the right model usually becomes clear from there.
Sources
Responses (0)
Sign in to join the conversation and share your thoughts.

Prompt & Circuit Editorial
Contributing Editor
Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.
Continue Reading

Train, Serve, and Deploy a Scikit-learn Model with FastAPI
Train a scikit-learn classifier, serve it behind a typed FastAPI endpoint, test it locally and deploy it with the FastAPI CLI. Code tested on scikit-learn 1.9 and FastAPI 0.159.

Linear Regression in Practice: Fitting, Evaluating and Checking Assumptions in Python
Linear regression is the baseline every regression project should start from. This guide covers the least-squares objective, gradient descent versus a direct solver, evaluation metrics and assumption checks, with tested NumPy, scikit-learn and statsmodels code.