Skip to main content

Execution for Apache Sparks

Spark execution describes how a query becomes work on a cluster: Catalyst builds logical and physical plans, the DAG scheduler cuts them into stages at shuffle boundaries, and each stage runs as parallel tasks over partitions. Reading these plans is the basis for all performance diagnosis.

Stage: Performance · Part of the Apache Spark roadmap · Last reviewed 19 November 2026

What to learn in Execution

Execution interview questions

What is a shuffle and why is it expensive?

A shuffle redistributes data across partitions so rows with the same key land together, as in groupBy, join or repartition. It writes intermediate files to disk and sends data over the network, so it costs I/O, serialisation and network time. Reducing shuffles is a main tuning goal.

How do you read a Spark query plan?

Run explain, or explain with the formatted mode, and read the physical plan from the bottom up. Look at the file scan for pushed filters and pruned columns, then at join types and Exchange nodes, which mark shuffles. The SQL tab in the Spark UI shows the same tree with metrics.

Where Execution fits in the Apache Spark roadmap

Execution belongs to the Performance stage (intermediate level). Tune it.

Back to the full Apache Spark roadmap