Execution for Apache Sparks
Spark execution describes how a query becomes work on a cluster: Catalyst builds logical and physical plans, the DAG scheduler cuts them into stages at shuffle boundaries, and each stage runs as parallel tasks over partitions. Reading these plans is the basis for all performance diagnosis.
Stage: Performance · Part of the Apache Spark roadmap · Last reviewed 19 November 2026
What to learn in Execution
- Read the output of explain and identify scans, filters, joins and exchanges
- Spot a shuffle in a plan and name the operation that caused it
- Use the Spark UI to find the stage and task that dominate a job's runtime
- Tell narrow dependencies from wide dependencies and relate them to stage boundaries
- Recognise data skew from uneven task durations in a stage
Execution interview questions
What is a shuffle and why is it expensive?
A shuffle redistributes data across partitions so rows with the same key land together, as in groupBy, join or repartition. It writes intermediate files to disk and sends data over the network, so it costs I/O, serialisation and network time. Reducing shuffles is a main tuning goal.
How do you read a Spark query plan?
Run explain, or explain with the formatted mode, and read the physical plan from the bottom up. Look at the file scan for pushed filters and pruned columns, then at join types and Exchange nodes, which mark shuffles. The SQL tab in the Spark UI shows the same tree with metrics.
Where Execution fits in the Apache Spark roadmap
Execution belongs to the Performance stage (intermediate level). Tune it.
Other units in this stage
Back to the full Apache Spark roadmap