Skip to main content

Batch for Data Engineers

Batch processing is moving and transforming data in scheduled chunks, such as nightly loads. This unit covers ETL versus ELT patterns, transformation with dbt, and distributed engines such as Spark for data too large for one machine. It is the core of most data pipelines in practice.

Stage: Ingestion and Processing · Part of the Data Engineer roadmap · Last reviewed 19 November 2026

What to learn in Batch

Batch interview questions

What is the difference between ETL and ELT?

ETL transforms data before loading it into the target, usually on a separate processing server. ELT loads raw data first and transforms it inside the warehouse using SQL. ELT is popular because warehouses scale compute well and raw data stays available for reprocessing.

How does an incremental load work?

The job remembers the highest value of a column such as updated_at from the last run, then pulls only rows beyond it and merges them into the target. This saves time versus full reloads but needs care for late-arriving rows and deletes.

Where Batch fits in the Data Engineer roadmap

Batch belongs to the Ingestion and Processing stage (intermediate level), which takes about 7 hours. Move and transform data.

Back to the full Data Engineer roadmap