Skip to main content

Streaming for Apache Sparks

Spark Structured Streaming is the engine's stream processing API, which treats a live source such as Kafka as an unbounded table and runs the same DataFrame queries on it incrementally. It sits after batch work in the roadmap and adds triggers, watermarks, output modes and checkpoints.

Stage: Streaming and ML · Part of the Apache Spark roadmap · Last reviewed 19 November 2026

What to learn in Streaming

Streaming interview questions

What does a watermark do in Structured Streaming?

A watermark tells Spark how late data can arrive, measured against the maximum event time seen. Spark uses it to decide when a window is final, emit results and discard old state. Events older than the watermark may be dropped, which keeps state size bounded.

How does Structured Streaming achieve fault tolerance?

Progress and state are saved to a checkpoint directory, recording source offsets and aggregation state. After a failure the query resumes from the last committed offsets. With a replayable source and an idempotent or transactional sink, this gives end-to-end exactly-once results.

Where Streaming fits in the Apache Spark roadmap

Streaming belongs to the Streaming and ML stage (intermediate level). Beyond batch.

Back to the full Apache Spark roadmap