Skip to main content

Ingestion for Databrickss

Databricks ingestion is the work of getting raw data into Delta tables, from files in cloud storage, databases and event streams. It sits in the second stage and feeds the bronze layer of a medallion pipeline, usually through Auto Loader, COPY INTO or Structured Streaming sources.

Stage: Ingestion and Transformation · Part of the Databricks roadmap · Last reviewed 19 November 2026

What to learn in Ingestion

Ingestion interview questions

How does Auto Loader know which files it has already processed?

Auto Loader records processed file paths in a checkpoint location, using RocksDB, so a restarted stream skips them. It can discover files by directory listing or by cloud file notifications, and the checkpoint is what makes ingestion incremental and idempotent.

When would you pick COPY INTO over Auto Loader?

COPY INTO suits modest volumes and scheduled batch loads where simple SQL is enough, since it is idempotent per file. Auto Loader is better for very large or continuously arriving file sets, because it scales file discovery and supports schema evolution.

Where Ingestion fits in the Databricks roadmap

Ingestion belongs to the Ingestion and Transformation stage (intermediate level). Medallion pipelines.

Back to the full Databricks roadmap