Tagged with

15 articles found

MinIO’s Maintenance Mode: The Architectural Toll of Free Infrastructure

The sudden halt of a core S3-compatible storage system exposes a harsh truth about the hidden costs of depending on ‘free’ open-source software.

#data-engineering#kubernetes#open-source

benchmark

Iceberg vs Delta Lake: Your Open Source Bet Might Be Cheaper Than You Think

When Apache Iceberg finished 18% faster and 61% cheaper than Databricks in our TPC-H benchmark, we realized the open core dilemma isn’t just philosophical, it’s financial.

#benchmark#data-engineering#lakehouse...

data-engineering

Data Quality Responsibility Is a Product Problem, Not an Engineering Failure

Why engineering teams are tired of cleaning up upstream messes that product managers should own from day one.

#data-engineering#data-governance#data-quality...

data-architecture

The Medallion Architecture Debate: Why Nobody Agrees on Silver vs. Gold

Exploring the hidden inconsistencies in how teams implement raw-silver-gold layering models, from transformation logic disagreements to the platinum layer controversy.

#data-architecture#data-engineering#databricks...

data-engineering

JSON in PySpark: The Performance Trap You’re Probably Falling For

Why writing large PySpark DataFrames as JSON to S3 is fundamentally flawed – and what you should do instead

#data-engineering#json#pyspark...

data-engineering

The Death of OSS dbt: Merging with Fivetran and the End of Open Source Trust?

The controversial merger between dbt Labs and Fivetran puts open source data tooling’s future in question

#data-engineering#dbt#fivetran...

apache-iceberg

Parquet Was Never Enough: Why Your Raw Data Lake Is a Data Swamp Waiting to Happen

While Parquet excels at storage efficiency, it lacks transactional integrity. Open table formats like Iceberg, Delta Lake, and Hudi are increasingly seen as essential for reliable data pipelines by adding ACID compliance and schema evolution.

#apache-iceberg#apache-parquet#data-architecture...

data-engineering

The Fivetran-dbt “Marriage of Convenience” That’s Quietly Declaring War on Snowflake

The merger between Fivetran and dbt Labs isn’t just consolidation, it’s a strategic power play to flip the data stack hierarchy and make warehouses the commodities, not the kings.

#data-engineering#databricks#dbt...

data-architecture

The Great Data Tooling Debate: SQL vs. Spark Ecosystem

Enterprise data teams are increasingly questioning the need for complex tooling stacks when SQL in platforms like Snowflake can handle most data transformations.

#data-architecture#data-engineering#snowflake...

ai-infrastructure

Why treating AI data as transactional tables beats unstructured blob chaos

#ai-infrastructure#data-engineering#gpu-optimization...

best-practices

ETL Testing: The Silent Killer of Data Engineering Projects

Why testing ETL pipelines remains the most overlooked and underestimated challenge in modern data architectures

#best-practices#data-engineering#data-pipelines...

apache-fluss

Is Fluss About to Make Kafka Obsolete for Lakehouse Real-Time?

Apache Fluss 0.8.0 introduces Iceberg support, enabling sub-second latency processing via Flink and positioning itself as the hot layer solution for Iceberg-based lakehouses.

#apache-fluss#data-engineering#flink...

data-engineering

Database Design: Integrity Clashes With Performance

The eternal database battle, seen through a librarian’s weary eyes, how to choose between clean data and fast queries

#data-engineering#data-modeling#database-design...

apache-iceberg

The Death of Data Partitioning? Clustering Takes Over in Modern Data Lakes

Emerging shift from traditional data partitioning to clustering techniques in Delta Lake, Iceberg, Snowflake, and Databricks, and what it means for data engineering best practices.

#apache-iceberg#data-engineering#data-lakes...

big-data

Java Streaming is Dead: Why Your 75GB CSV Ingestion Strategy Belongs in 2015

A brutal case study comparing Java streaming approaches against modern tools like DuckDB and Spark for massive data ingestion, revealing why traditional methods are costing you time and sanity.

#big-data#data-engineering#duckdb...