Tagged with

19 articles found

Snowflake OpenFlow: The Native CDC Savior Is Still On Training Wheels

Snowflake’s OpenFlow promises seamless CDC from Oracle and SQL Server, but early adopters reveal performance and usability gaps that make it a tough sell against established tools.

#cdc#data-engineering#etl...

cost-optimization

The Databricks Exit: Why Broke Teams Are Building Better Data Stacks on Kubernetes

Budget cuts are forcing teams off Azure Databricks and onto open-source alternatives. The twist? They’re discovering more flexible, composable architectures that might be superior, even if they could afford the managed platforms.

#cost-optimization#data-engineering#databricks...

data-engineering

Your Vendor’s Excel Files Are a Silent Data Pipeline Killer

Why inconsistent vendor spreadsheets create a hidden crisis in data engineering and how validation-driven ingestion is the only real solution.

#data-engineering#data-quality#etl...

aws-glue

The Persistent Nightmare of Datetime Handling in Data Engineering

Despite decades of computing progress, datetime formatting remains a major pain point for data engineers, leading to bugs, pipeline breaks, and widespread frustration across systems and timezones.

#aws-glue#data-engineering#datetime...

data-engineering

MinIO’s Maintenance Mode: The Architectural Toll of Free Infrastructure

The sudden halt of a core S3-compatible storage system exposes a harsh truth about the hidden costs of depending on ‘free’ open-source software.

#data-engineering#kubernetes#open-source

benchmark

Iceberg vs Delta Lake: Your Open Source Bet Might Be Cheaper Than You Think

When Apache Iceberg finished 18% faster and 61% cheaper than Databricks in our TPC-H benchmark, we realized the open core dilemma isn’t just philosophical, it’s financial.

#benchmark#data-engineering#lakehouse...

data-engineering

Data Quality Responsibility Is a Product Problem, Not an Engineering Failure

Why engineering teams are tired of cleaning up upstream messes that product managers should own from day one.

#data-engineering#data-governance#data-quality...

data-architecture

The Medallion Architecture Debate: Why Nobody Agrees on Silver vs. Gold

Exploring the hidden inconsistencies in how teams implement raw-silver-gold layering models, from transformation logic disagreements to the platinum layer controversy.

#data-architecture#data-engineering#databricks...

data-engineering

JSON in PySpark: The Performance Trap You’re Probably Falling For

Why writing large PySpark DataFrames as JSON to S3 is fundamentally flawed – and what you should do instead

#data-engineering#json#pyspark...

data-engineering

The Death of OSS dbt: Merging with Fivetran and the End of Open Source Trust?

The controversial merger between dbt Labs and Fivetran puts open source data tooling’s future in question

#data-engineering#dbt#fivetran...

apache-iceberg

Parquet Was Never Enough: Why Your Raw Data Lake Is a Data Swamp Waiting to Happen

While Parquet excels at storage efficiency, it lacks transactional integrity. Open table formats like Iceberg, Delta Lake, and Hudi are increasingly seen as essential for reliable data pipelines by adding ACID compliance and schema evolution.

#apache-iceberg#apache-parquet#data-architecture...

data-engineering

The Fivetran-dbt “Marriage of Convenience” That’s Quietly Declaring War on Snowflake

The merger between Fivetran and dbt Labs isn’t just consolidation, it’s a strategic power play to flip the data stack hierarchy and make warehouses the commodities, not the kings.

#data-engineering#databricks#dbt...

data-architecture

The Great Data Tooling Debate: SQL vs. Spark Ecosystem

Enterprise data teams are increasingly questioning the need for complex tooling stacks when SQL in platforms like Snowflake can handle most data transformations.

#data-architecture#data-engineering#snowflake...

ai-infrastructure

Why treating AI data as transactional tables beats unstructured blob chaos

#ai-infrastructure#data-engineering#gpu-optimization...

best-practices

ETL Testing: The Silent Killer of Data Engineering Projects

Why testing ETL pipelines remains the most overlooked and underestimated challenge in modern data architectures

#best-practices#data-engineering#data-pipelines...

apache-fluss

Is Fluss About to Make Kafka Obsolete for Lakehouse Real-Time?

Apache Fluss 0.8.0 introduces Iceberg support, enabling sub-second latency processing via Flink and positioning itself as the hot layer solution for Iceberg-based lakehouses.

#apache-fluss#data-engineering#flink...

data-engineering

Database Design: Integrity Clashes With Performance

The eternal database battle, seen through a librarian’s weary eyes, how to choose between clean data and fast queries

#data-engineering#data-modeling#database-design...

apache-iceberg

The Death of Data Partitioning? Clustering Takes Over in Modern Data Lakes

Emerging shift from traditional data partitioning to clustering techniques in Delta Lake, Iceberg, Snowflake, and Databricks, and what it means for data engineering best practices.

#apache-iceberg#data-engineering#data-lakes...

big-data

Java Streaming is Dead: Why Your 75GB CSV Ingestion Strategy Belongs in 2015

A brutal case study comparing Java streaming approaches against modern tools like DuckDB and Spark for massive data ingestion, revealing why traditional methods are costing you time and sanity.

#big-data#data-engineering#duckdb...