BANANDRE
NO ONE CARES ABOUT CODE

Navigation

HomeCategories

Categories

Artificial Intelligence(406)
Software Development(213)
Software Architecture(190)
Data Engineering(110)
Engineering Management(56)
Enterprise Architecture(35)
Product Management(27)
tech(1)

Tagged with

#pyspark

5 articles found

DuckDB Is Eating the CSV-to-Parquet Pipeline
csv
Featured

DuckDB Is Eating the CSV-to-Parquet Pipeline

How to convert massive 80GB CSV files to Parquet without melting your RAM, and why DuckDB has become the default tool for data engineers fighting memory constraints.

#csv#duckdb#pandas...
Read More
The Death of PySpark? Why SQL Rules the Gold Layer
databricks

The Death of PySpark? Why SQL Rules the Gold Layer

Data pipelines are quietly abandoning Spark processing for final aggregation layers. The shift isn’t about performance, it’s about who actually maintains the code.

#databricks#pyspark
Read More
Data Modeling Is Killing Your PySpark Performance, Not Join Optimization
dag-complexity

Data Modeling Is Killing Your PySpark Performance, Not Join Optimization

A technical post-mortem on why a 50k-row table brought down a Databricks cluster, exposing the dangerous gap between software engineering instincts and distributed data architecture

#dag-complexity#data-modeling#databricks...
Read More
The Persistent Nightmare of Datetime Handling in Data Engineering
aws-glue

The Persistent Nightmare of Datetime Handling in Data Engineering

Despite decades of computing progress, datetime formatting remains a major pain point for data engineers, leading to bugs, pipeline breaks, and widespread frustration across systems and timezones.

#aws-glue#data-engineering#datetime...
Read More
JSON in PySpark: The Performance Trap You’re Probably Falling For
data-engineering

JSON in PySpark: The Performance Trap You’re Probably Falling For

Why writing large PySpark DataFrames as JSON to S3 is fundamentally flawed – and what you should do instead

#data-engineering#json#pyspark...
Read More
BANANDRE
NO ONE CARES ABOUT CODE

Connect

2026 BANANDRE
Privacy PolicyTermsImpressum
Built with 🍌