Tagged with

3 articles found

Ditching Pandas for DuckDB: The Memory Crisis Hiding in Your Validation Pipeline

Evaluating the technical cost-benefit of replacing Pandas memory-intensive operations with DuckDB direct-to-cloud queries for large-scale validation tasks on S3.

#arrow#data-validation#duckdb...

API design

The Cold Storage Lie: Why Your API Might Be Fine Querying S3 Directly

Fintech systems are hemorrhaging money on cold data storage. The conventional wisdom says you need Redshift or Athena, but the math, and the architecture, tells a more complicated story.

#API design#cold storage#Fintech...

data-engineering

JSON in PySpark: The Performance Trap You’re Probably Falling For

Why writing large PySpark DataFrames as JSON to S3 is fundamentally flawed – and what you should do instead

#data-engineering#json#pyspark...