Transformation for Databrickss
Databricks transformation is the step that turns raw bronze data into cleaned silver tables and business-ready gold tables using Spark SQL, PySpark and declarative pipelines. It sits after ingestion and applies joins, deduplication, type fixes and aggregations while keeping the output as governed Delta tables.
Stage: Ingestion and Transformation · Part of the Databricks roadmap · Last reviewed 19 November 2026
What to learn in Transformation
- Build a bronze, silver and gold pipeline with clear responsibilities for each layer
- Deduplicate and upsert records into a silver table using MERGE INTO
- Write a PySpark transformation using select, withColumn, join and window functions
- Declare tables and expectations in a Lakeflow declarative pipeline
- Pick between a view, a materialized view and a table for a gold output
Transformation interview questions
How do you upsert changed records into a Delta table?
Use MERGE INTO with the target table, a source of new data, a match condition on the business key, and WHEN MATCHED UPDATE and WHEN NOT MATCHED INSERT clauses. Delta applies it as one atomic transaction, so readers never see half the changes.
What is the medallion architecture?
It is a layered design where bronze holds raw ingested data, silver holds cleaned and conformed data, and gold holds aggregated tables shaped for reports or models. Each layer is a Delta table, which makes lineage clear and lets you rebuild later layers from earlier ones.
Where Transformation fits in the Databricks roadmap
Transformation belongs to the Ingestion and Transformation stage (intermediate level). Medallion pipelines.
Other units in this stage
Back to the full Databricks roadmap