Delta Lake 4.3 Just Killed Your Favorite Overwrite Hack (And You Should Be Grateful)

The replaceWhere Tax: Why the Old Way Was Broken

Let’s be precise about what replaceWhere required. You’d write your DataFrame with a predicate specifying which partition or logical range should be replaced:

updates.write \
    .mode("overwrite") \
    .option("replaceWhere", "order_date >= '2026-06-01' AND region = 'EMEA'") \
    .saveAsTable("testing.default.orders")

That predicate is a second source of truth. It’s not derived from the data, it’s a separate statement of intent that you must keep synchronized with whatever your upstream job produces. Each delayed partition or new business region is another chance for the predicate to be wrong. And when it’s wrong, the failure is rarely loud. You get stale data. Or duplicates. Or, in the worst case, a partition that silently disappears because an upstream failure produced an empty DataFrame and replaceWhere dutifully deleted the matching range.

The community has been circling this problem for years. The limitations of raw Parquet-based lakes are well documented, and Delta Lake was supposed to be the fix. But replaceWhere always felt like a Band-Aid, functional, but fragile.

replaceUsing: The Predicate-Free Overwrite

Here’s what the new API looks like in practice:

updates = spark.read.table("testing.default.orders_updates")

updates.write \
    .mode("overwrite") \
    .option("replaceUsing", "order_date, region") \
    .saveAsTable("testing.default.orders")

That’s it. You specify the columns that identify a row, and Delta handles the rest. Rows in orders where (order_date, region) matches a pair in updates get replaced. Rows where the pair doesn’t match stay untouched. Source rows containing a brand-new pair get inserted.

No predicate to sync. No second source of truth. If your upstream job suddenly starts producing data for a new region, the pipeline handles it correctly without a single line of code changing.

Important caveat: this is not partition overwrite. The matching columns don’t need to be partition columns. It works on partitioned tables, non-partitioned tables, and liquid-clustered tables alike. On Databricks, full functionality requires Runtime 17.2+, versions 16.3 through 17.1 still require a partitioned table with all partition columns in the match.

The NULL Trap: Why replaceOn Exists

Here’s where things get interesting. replaceUsing treats NULL as just another value, or more accurately, as nothing. Like JOIN USING, it will never match two NULLs. So if your region column is NULL on both the target and source rows, the old row never gets replaced. You silently end up with a duplicate.

Enter replaceOn, which gives you a boolean condition over aliased source and target tables:

updates.alias("s") \
    .write \
    .mode("overwrite") \
    .option("targetAlias", "t") \
    .option("replaceOn", "s.order_date <=> t.order_date AND s.region <=> t.region") \
    .saveAsTable("testing.default.orders")

The <=> operator is NULL-safe equality. Two NULLs actually match. The old row gets replaced, no duplicate appears.

If your keys are never NULL, stick with replaceUsing, it’s simpler and recommended. But for anyone working with semi-structured or incomplete data, replaceOn is the safety net that prevents a whole class of subtle bugs.

The Behavioral Difference That Actually Matters

Here’s the detail that should make you rethink your existing replaceWhere pipelines: empty source handling.

With replaceWhere, an empty source DataFrame will dutifully delete the rows in the matching predicate range. Your upstream job fails silently, produces no data, and your pipeline wipes a partition. The data is gone. Good luck explaining that one on the Monday morning call.

With replaceUsing and replaceOn, an empty source deletes nothing. The write is effectively a no-op. If a failure upstream ever produces an empty dataset, your table stays intact. That alone is a reason to migrate.

There are constraints worth knowing before you refactor:

• You can’t combine replaceUsing or replaceOn with replaceWhere, partitionOverwriteMode, or overwriteSchema in Python or Scala
• All three (including replaceWhere) now reject subquery predicates
• Databricks SQL forms (REPLACE USING in Runtime 16.3+, REPLACE ON in 17.1+) have been available for a while, but Python and Scala DataFrame support requires Databricks Runtime 18.2+

The Unity Catalog Shift: What Changes Under the Hood

The second major change in 4.3 is structural and largely invisible, which is precisely why you should care. Delta Spark now routes all operations on catalog-managed tables through Unity Catalog’s Delta REST APIs. In 4.2, commits were catalog-coordinated. In 4.3, table loads, CREATE, CTAS, REPLACE, and metadata-changing writes all go through the same validated path.

Three guarantees come out of this:

1. Server-side commit validation rejects malformed or conflicting commits before they’re final. A bad write can’t corrupt a table anymore.
2. Server-declared table features let the catalog tell engines what a new table supports, rather than each engine deciding independently.
3. Intent-based metadata updates mean the engine declares what it wants to change, and the catalog validates and applies it, instead of writing metadata directly.

The practical outcome: DuckDB, Flink, Trino, and Spark all read and write the same tables under the same set of rules. That’s a significant reduction in the operational risk of multi-engine access, which historically has been one of the biggest operational taxes of open table formats.

UniForm Gets Its Act Together

The other quiet upgrade worth noticing: UniForm, which syncs Iceberg metadata with Delta commits, now handles two issues that previously disqualified it for serious use.

Atomic and incremental transformation. Large commits now convert to Iceberg metadata within the Delta transaction itself, not after it. That closes the consistency gap where an Iceberg reader could see a stale snapshot during bulk commits. Incremental conversion means only the changed portion of the Delta log is rebuilt, making UniForm viable for tables with long histories.

IcebergCompatV3 (experimental) lets you use deletion vectors and UniForm simultaneously. Previously it was an either/or: fast deletes and merges or Iceberg reader compatibility, never both. Now you can enable both:

CREATE TABLE testing.default.orders (
  order_date DATE,
  region STRING,
  order_id STRING,
  amount DECIMAL(12, 2)
)
USING DELTA
TBLPROPERTIES (
  'delta.enableIcebergCompatV3'          = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg',
  'delta.feature.catalogManaged'         = 'supported',
  'delta.enableDeletionVectors'          = 'true'
);

One gotcha for your upgrade notes: in UniForm tables with deletion vectors, Iceberg’s DataFile.recordCount now reports the physical row count before deletion vectors are applied. Any downstream process reading that value needs to apply the vectors to get logical counts. Nothing breaks loudly, the numbers are just wrong.

Streaming Gets the Same Treatment

Structured Streaming and Change Data Capture now work with catalog-managed Delta tables from Apache Spark. Batch CDC comes via a CHANGES clause that respects deletion vectors:

SET spark.databricks.delta.changelogV2.enabled = true;

-- Replay all changes from a known version
SELECT * FROM testing.default.orders CHANGES FROM VERSION 0;

-- Or from a specific point in time
SELECT * FROM testing.default.orders CHANGES FROM TIMESTAMP '2026-06-01 00:00:00';

The CHANGES syntax is the same one Apache Spark 4.2 standardized across all connectors, so incremental-read logic written with it stays portable.

Small Changes, Real Impact

Beyond the headline features, 4.3 has details that quietly improve your day:

• V2 checkpoints default to 50,000 actions per sidecar file. Sidecars are split into multiple files, and checkpoint writing runs in parallel by default. Large tables get extra throughput on a path you never had to configure.
• Variant column stats at write time. Delta Spark now collects min/max values for Variant columns, enabling data skipping on tables with those columns. Semi-structured columns stop being a blocker for file pruning.
• Implicit type casting for DataFrame writes by name. Except for save() and saveAsTable().mode("overwrite"), writing by name now applies Spark’s implicit cast rules to match source values to the target schema, consistent with SQL INSERT BY NAME.
• Faster Kernel diagnostics. tableSizeBytes and numFiles are computed incrementally from version checksums rather than full log replay. Kernel can also open tables with missing or stale _last_checkpoint pointers via log scanning.

Before You Upgrade: Breaking Changes

Three things will bite you if you’re not paying attention:

1. MERGE INTO with an empty schema now errors when mergeSchema=false. Previously it overwrote the target schema with the source schema without warning. Set delta.schemaAutoMerge.enabled=true or align your schemas first.
2. Iceberg DataFile.recordCount changes for UniForm + deletion vector tables, as described above.
3. Kernel’s enableVariantShredding is now on by default under the variantShredding-preview feature. If you need production-ready functionality, explicitly request it with delta.feature.variantShredding=supported.

What I’d Actually Do This Week

Find the pipeline where replaceWhere has caused you grief and rewrite it with replaceUsing. It’s a small change with immediate correctness benefits, and the empty-source behavior alone makes retries safer. Then look for any table in your ecosystem stuck in the deletion-vector-or-UniForm tradeoff, IcebergCompatV3 takes that tradeoff off the table.

The rest of 4.3 is infrastructure: it makes things better whether you interact with it or not.

One more note before you plan the upgrade: Delta Lake 4.4 shipped in August 2026 with Apache Spark 4.2 as the default. If you’re planning an upgrade now, plan for 4.4 and treat this article as a roadmap of what landed on the way there.

For teams weighing their options, the competition between Delta Lake and Apache Iceberg keeps heating up. If you’re evaluating whether managed Delta Lake is worth it, the cost of building your own lakehouse is a harder sell than it used to be. And if you’re wondering whether the ecosystem will remain Spark-centric, the Polars vs. Spark debate has a delta-shaped answer.

The takeaway: Delta Lake 4.3 isn’t a flashy release. It’s a quality-of-life release that fixes the class of bugs data engineers have learned to live with. replaceUsing is the headline, but the catalog-backed validation might be the real foundation being laid.