Operations for Databrickss
Databricks operations covers keeping pipelines healthy after launch: reading Spark UI and job run logs, diagnosing slow or failed runs, and tuning storage and compute. It sits in the advanced stage and uses tools such as OPTIMIZE, liquid clustering, system tables and cluster sizing to control performance and cost.
Stage: Operations and Governance · Part of the Databricks roadmap · Last reviewed 19 November 2026
What to learn in Operations
- Read the Spark UI to find skewed tasks, spills and long shuffles
- Run OPTIMIZE and VACUUM and explain what each does to files
- Decide when liquid clustering or partitioning fits a table's query pattern
- Query system tables to see billing usage and job run history
- Right-size compute and use autoscaling without overpaying
Operations interview questions
A Spark job is slow because one task runs far longer than the rest. What is likely wrong?
That pattern points to data skew, where one key holds a disproportionate share of rows. Check the Spark UI stage view, then try salting the key, enabling adaptive query execution skew handling, or filtering the hot key out and handling it separately.
What does VACUUM do on a Delta table and what is the risk?
VACUUM deletes data files no longer referenced by the transaction log and older than the retention period, which defaults to seven days. Setting retention too short can break time travel and long-running readers or streams that still need those files.
Where Operations fits in the Databricks roadmap
Operations belongs to the Operations and Governance stage (advanced level). Run safely.
Other units in this stage
Back to the full Databricks roadmap