Tagged with

4 articles found

Your Spark Pipelines Are Still Writing Code Like It’s 2015. SDP Is Here to Fix That.

Spark Declarative Pipelines are becoming native to Apache Spark 4.1. Here’s what changes for data engineers, and why the open-source community is finally paying attention.

#apache spark#data pipelines#Delta Live Tables

apache spark

Delta Lake 4.3 Just Killed Your Favorite Overwrite Hack (And You Should Be Grateful)

replaceUsing and replaceOn finally give Delta Lake a selective overwrite primitive that isn’t replaceWhere. Here’s what changes for your ETL pipelines.

#apache spark#Data Engineering#delta lake

apache spark

You Migrated from Spark to dbt. Your Incremental Models Got Slower. Here’s Why.

A real-world account of migrating from Spark-based ETL to dbt ELT on Redshift, highlighting the surprising performance drawbacks of Redshift Serverless for incremental models compared to provisioned Redshift.

#apache spark#Data Engineering#dbt...

apache spark

The Rust Wave: Are We Finally Moving Beyond Spark and Java?

Exploring how Rust-based data engines are delivering massive performance gains by cutting through JVM overhead and garbage collection bottlenecks.

#apache spark#big data#Data Engineering