Native Auto CDC in Apache Spark 4.2.0: The end of manual MERGE statements

I have spent more hours than I care to admit untangling imperative MERGE statements. Managing slowly changing dimensions usually starts out simple enough. Then a few late-arriving records hit the pipeline, and a straightforward script turns into a brittle web of edge cases.

On July 14, 2026, the Apache Spark community dropped version 4.2.0, closing out over 1,700 JIRA tickets. Rather than just patching bugs, this release elevates Change Data Capture (CDC) to a first-class engine primitive. We are finally stepping away from fragmented, storage-specific operations. The engine is actually doing the heavy lifting for us.

Declarative pipelines and Auto CDC

Tracked under SPARK-56249, Auto CDC brings declarative Slowly Changing Dimensions (SCD) Type 1 and Type 2 processing to open-source Spark Declarative Pipelines. We no longer have to explicitly code the logic to avoid table corruption. By shifting to declarative definitions, we just state what the end state of the pipeline should look like.

If a record arrives three days late, or an update only contains partial fields while ignoring NULL values, Auto CDC figures out the reconciliation automatically. Engineering teams can spend their cycles experimenting with new streaming data products instead of babysitting batch job failures.

Standardizing the lakehouse

Querying CDC used to mean learning a specific dialect depending on whether your storage layer ran on Delta Lake, Apache Iceberg, or Apache Hudi. The new SQL CHANGES clause, tracked under SPARK-55668 and SPARK-55948, standardizes this across all these formats via a unified Data Source V2 (DSv2) Changelog interface.

Because the Catalyst optimizer now natively understands CDC post-processing, it filters out copy-on-write carry-over rows and computes net changes directly within the engine. Removing format-specific lock-in means we can start mixing and matching storage layers based on workload needs rather than arbitrary vendor constraints.

Zero-file read-time CDC

Pairing Spark 4.2.0 with Delta Lake 4.4.0 unlocks zero-file read-time CDC. Previously, tracking changes meant writing physical CDC files, which often added a 1.2x to 2x write overhead. Now, the engine dynamically reconstructs insert, delete, and update streams exactly when you read them, using existing metadata columns like row ID and commit version.

Recent Databricks telemetry showed that across 1.5 billion CDC calls, 70 percent of tables no longer needed physical CDC files. That is a concrete reduction in write amplification. Legacy MERGE codebases will carry some migration debt, and you still have to manage tombstone garbage collection carefully. Even so, the tradeoff is a leaner infrastructure footprint that frees up compute and storage budgets for other projects.

Bitemporal tracking and AI workflows

We are also seeing beta updates to the Auto CDC API that introduce bitemporal tracking. By monitoring changes across both business time and system time, organizations can build strict regulatory auditing systems that reflect reality at any given historical moment. You could theoretically rewind your data state to see exactly what an algorithm knew on a specific Tuesday last year.

Then there is the integration with AI coding assistants. Because we are moving toward declarative SQL and Python pipelines, we change how we interact with LLMs. Rather than asking a coding assistant to write a convoluted imperative script that might corrupt the JVM, these tools can safely generate and iterate on pipeline code via Spark Connect. The assistant defines the state, and Spark figures out the execution. We might soon see a workflow where developers just sketch out data lineage requirements in plain English, and the assistant reliably compiles the declarative pipeline underneath.