Machine Learning for Apache Sparks
Spark MLlib is the machine learning library of Apache Spark, offering distributed algorithms for classification, regression, clustering and recommendation, plus feature transformers and Pipelines built on DataFrames. It is used when training data is too large for one machine's memory.
Stage: Streaming and ML · Part of the Apache Spark roadmap · Last reviewed 19 November 2026
What to learn in Machine Learning
- Assemble features with VectorAssembler, StringIndexer and OneHotEncoder inside a Pipeline
- Train a logistic regression or random forest model on a Spark DataFrame
- Evaluate a model with BinaryClassificationEvaluator or RegressionEvaluator on held-out data
- Tune hyperparameters using CrossValidator and ParamGridBuilder
- Save a fitted PipelineModel and load it again for batch scoring
Machine Learning interview questions
What is a Pipeline in Spark MLlib?
A Pipeline chains transformers and estimators, such as an indexer, an assembler and a classifier, into one object. Calling fit trains each stage in order and returns a PipelineModel, which applies the same steps at prediction time. This avoids mismatched preprocessing between training and scoring.
When would you pick MLlib over scikit-learn?
Choose MLlib when the training data does not fit in one machine's memory or already lives in a Spark cluster. For data that fits on a single node, scikit-learn offers a wider range of algorithms and is usually simpler and faster, so use it first.
Where Machine Learning fits in the Apache Spark roadmap
Machine Learning belongs to the Streaming and ML stage (intermediate level). Beyond batch.
Other units in this stage
Back to the full Apache Spark roadmap