Ingestion for Databrickss
Databricks ingestion is the work of getting raw data into Delta tables, from files in cloud storage, databases and event streams. It sits in the second stage and feeds the bronze layer of a medallion pipeline, usually through Auto Loader, COPY INTO or Structured Streaming sources.
Stage: Ingestion and Transformation · Part of the Databricks roadmap · Last reviewed 19 November 2026
What to learn in Ingestion
- Load new files incrementally from cloud storage using Auto Loader with cloudFiles
- Choose between COPY INTO and Auto Loader for a given file arrival pattern
- Configure schema inference and schema evolution so new columns do not break a load
- Read from a Kafka topic with Structured Streaming into a bronze table
- Handle malformed records with a rescued data column
Ingestion interview questions
How does Auto Loader know which files it has already processed?
Auto Loader records processed file paths in a checkpoint location, using RocksDB, so a restarted stream skips them. It can discover files by directory listing or by cloud file notifications, and the checkpoint is what makes ingestion incremental and idempotent.
When would you pick COPY INTO over Auto Loader?
COPY INTO suits modest volumes and scheduled batch loads where simple SQL is enough, since it is idempotent per file. Auto Loader is better for very large or continuously arriving file sets, because it scales file discovery and supports schema evolution.
Where Ingestion fits in the Databricks roadmap
Ingestion belongs to the Ingestion and Transformation stage (intermediate level). Medallion pipelines.
Other units in this stage
Back to the full Databricks roadmap