Skip to main content

Basics for Apache Sparks

Spark basics covers the architecture and core abstractions of the engine: a driver program that schedules tasks on executors, RDDs as the low-level distributed collection, and DataFrames with Spark SQL as the structured API. It is the first stage of the roadmap, because every later topic builds on lazy transformations and actions.

Stage: Core Spark · Part of the Apache Spark roadmap · Last reviewed 19 November 2026

What to learn in Basics

Basics interview questions

Why are Spark transformations lazy?

Spark records transformations as a lineage graph and only runs it when an action such as count, collect or write is called. This lets the Catalyst optimizer see the whole plan, prune columns, push filters down and combine steps, and it lets Spark recompute lost partitions from lineage.

What is the difference between an RDD and a DataFrame?

An RDD is a distributed collection of objects with no schema, so Spark cannot optimise inside your functions. A DataFrame has named, typed columns, so Catalyst and the Tungsten engine can plan and execute it more efficiently. DataFrames are the default choice, while RDDs give low-level control.

Where Basics fits in the Apache Spark roadmap

Basics belongs to the Core Spark stage (beginner level), which takes about 23 hours. The programming model.

Back to the full Apache Spark roadmap