Streaming for Apache Sparks
Spark Structured Streaming is the engine's stream processing API, which treats a live source such as Kafka as an unbounded table and runs the same DataFrame queries on it incrementally. It sits after batch work in the roadmap and adds triggers, watermarks, output modes and checkpoints.
Stage: Streaming and ML · Part of the Apache Spark roadmap · Last reviewed 19 November 2026
What to learn in Streaming
- Read a Kafka topic with readStream and parse the value column into typed fields
- Aggregate events over tumbling and sliding windows keyed by event time
- Set a watermark and explain how it bounds state and drops late data
- Pick an output mode of append, update or complete for a given query
- Configure a checkpoint location so a query restarts without losing progress
Streaming interview questions
What does a watermark do in Structured Streaming?
A watermark tells Spark how late data can arrive, measured against the maximum event time seen. Spark uses it to decide when a window is final, emit results and discard old state. Events older than the watermark may be dropped, which keeps state size bounded.
How does Structured Streaming achieve fault tolerance?
Progress and state are saved to a checkpoint directory, recording source offsets and aggregation state. After a failure the query resumes from the last committed offsets. With a replayable source and an idempotent or transactional sink, this gives end-to-end exactly-once results.
Where Streaming fits in the Apache Spark roadmap
Streaming belongs to the Streaming and ML stage (intermediate level). Beyond batch.
Other units in this stage
Back to the full Apache Spark roadmap