Data Engineering - Banandre

Delta Lake 4.3 Just Killed Your Favorite Overwrite Hack (And You Should Be Grateful)

replaceUsing and replaceOn finally give Delta Lake a selective overwrite primitive that isn’t replaceWhere. Here’s what changes for your ETL pipelines.

#apache spark#Data Engineering#delta lake

best practices

dbt Best Practices Are a Floor, Not a Ceiling, Here’s Where to Build Higher

"{{ doc(‘customer_id’) }}"

#best practices#Data Engineering#Data Modeling...

developer-productivity

Fabric’s Low-Code Obsession Is Driving Data Engineers Absolutely Insane

Microsoft Fabric prioritizes ease-of-use over developer productivity. Here’s why that tradeoff is creating a silent revolt in enterprise data teams.

#developer-productivity#low-code

Data Engineering

Microsoft Fabric’s Developer Experience Crisis: When Clicking Isn’t Engineering

Why Microsoft Fabric’s GUI-first approach is alienating data engineers and what it means for the future of data platforms

#Data Engineering#data platforms#developer experience...

AI observability

Feeding Spreadsheets to GPT Won’t Save Your Forecast

LLMs are quietly rewriting how data scientists do predictive modeling and anomaly detection. But the real wins aren’t where most people are looking.

#AI observability#anomaly detection#data science...

compensation

The Salary Spreadsheet That’s Terrifying Tech Companies: Inside Data Engineering’s Pay Transparency Revolution

Community-driven salary datasets are reshaping how data engineers negotiate compensation. Here’s what the numbers reveal.

#compensation#data engineering careers#pay equity...

Airflow Is No Longer Just a Scheduler. It’s a Platform. The Hackathon Proves It.

The Beyond the DAG hackathon revealed Apache Airflow is evolving far beyond data pipelines, into AI agents, retro games, and general-purpose automation. Here’s what it means.

Data Engineering

ETL Pipeline Testing Strategies That Actually Scale

Real-world ETL testing approaches for massive datasets, incremental loads, and changing transformation logic.

#Data Engineering#data quality#ETL Testing...

Acquisition

The DuckDB Dilemma: AWS Just Bought the Team, But Who Really Owns the Duck?

DuckLabs is joining AWS. The open-source promises are reassuring, but history says otherwise. Here’s what the acquisition really means for DuckDB’s future.

#Acquisition#aws#Data Engineering...

Kafka Isn’t a Message Queue, Stop Treating It Like One

Kafka is a distributed commit log, not a task queue. Here’s what that actually means for your architecture, and where event-driven designs go wrong.

dlt

dlt vs. Hand-Rolled Python: Stop Re-Inventing the Pagination Wheel

Comparing dlt against custom Python ETL scripts for low-volume, multi-source ingestion workflows, and why your ‘simple’ script isn’t as simple as you think.

#dlt#python

data lake

S3 for Everything: The Universal Data Lake Trend That Refuses to Die

Turbopuffer, Neon, and Cursor are pushing ‘store everything in S3.’ But does the latency math work for your workloads?

#data lake#object storage

Category:

Delta Lake 4.3 Just Killed Your Favorite Overwrite Hack (And You Should Be Grateful)

dbt Best Practices Are a Floor, Not a Ceiling, Here’s Where to Build Higher

Fabric’s Low-Code Obsession Is Driving Data Engineers Absolutely Insane

Microsoft Fabric’s Developer Experience Crisis: When Clicking Isn’t Engineering

Feeding Spreadsheets to GPT Won’t Save Your Forecast

The Salary Spreadsheet That’s Terrifying Tech Companies: Inside Data Engineering’s Pay Transparency Revolution

Airflow Is No Longer Just a Scheduler. It’s a Platform. The Hackathon Proves It.

ETL Pipeline Testing Strategies That Actually Scale

The DuckDB Dilemma: AWS Just Bought the Team, But Who Really Owns the Duck?

Kafka Isn’t a Message Queue, Stop Treating It Like One

dlt vs. Hand-Rolled Python: Stop Re-Inventing the Pagination Wheel

S3 for Everything: The Universal Data Lake Trend That Refuses to Die

Delta Lake 4.3 Just Killed Your Favorite Overwrite Hack (And You Should Be Grateful)

dbt Best Practices Are a Floor, Not a Ceiling, Here’s Where to Build Higher

Fabric’s Low-Code Obsession Is Driving Data Engineers Absolutely Insane

Microsoft Fabric’s Developer Experience Crisis: When Clicking Isn’t Engineering

Feeding Spreadsheets to GPT Won’t Save Your Forecast

The Salary Spreadsheet That’s Terrifying Tech Companies: Inside Data Engineering’s Pay Transparency Revolution

Airflow Is No Longer Just a Scheduler. It’s a Platform. The Hackathon Proves It.

ETL Pipeline Testing Strategies That Actually Scale

The DuckDB Dilemma: AWS Just Bought the Team, But Who Really Owns the Duck?

Kafka Isn’t a Message Queue, Stop Treating It Like One

dlt vs. Hand-Rolled Python: Stop Re-Inventing the Pagination Wheel

S3 for Everything: The Universal Data Lake Trend That Refuses to Die