Every data engineer hits the same wall eventually. You’re building your third API-to-warehouse pipeline this quarter, and that fetch_with_retry() function you wrote last project is staring at you from your GitHub history. The siren song of “it’s just a simple script” gets louder with each new source you need to integrate.
Then someone mentions dlt, and suddenly you’re at a crossroads: keep hand-rolling your ingestion logic in Python, or trust a declarative tool to handle the boring stuff?
The answer isn’t as obvious as the dlt evangelists would have you believe, but it’s also not as complicated as the “I’ll just write my own” crowd pretends.
What dlt Actually Solves (and What It Doesn’t)
Let’s start with the uncomfortable truth: dlt is genuinely good at the stuff nobody wants to write. The Reddit consensus from engineers running it in production is overwhelmingly positive, and the reasons are remarkably consistent:
- Automatic schema evolution – When your API vendor adds a new field (they always do), dlt handles it without a deployment
- Nested JSON handling – It creates child tables with parent IDs automatically, which is exactly what you’d do manually but with about 200 fewer lines of code
- Built-in pagination – REST APIs with cursor-based pagination stop being a “fun” afternoon project
- Load timestamps – Every row gets a
_dlt_load_idwithout you writing a single line of instrumentation code - Auth handling – OAuth flows, API keys, token refreshes, all handled with configuration rather than ceremony
For the low-volume, full-load workflows described in the original question, relational databases and REST APIs with modest data volumes, dlt genuinely shines. You’re not paying a performance penalty for the abstraction because your bottleneck isn’t throughput, it’s the sheer drudgery of writing boilerplate.
The “It’s Great Until…” Problem
Here’s where the conversation gets interesting. The most common pushback against dlt isn’t that it’s bad, it’s that it has a ceiling.
One engineer’s experience sums it up: dlt handles 95% of cases perfectly, but when you hit an API that only serves one record at a time (baseurl/api/id with 30,000 new IDs daily), you’ll want an async solution instead. dlt isn’t built for that kind of pathological endpoint design, and forcing it will make you miserable.
This is the classic declarative tool dilemma. The tool handles the common case elegantly, but the moment you need something genuinely unusual, you’re either fighting the framework’s abstractions or writing a custom solution that lives outside the tool anyway.
The Hidden Cost of Custom Python (It’s Not What You Think)
The “I’ll just write a script” argument usually rests on three pillars: flexibility, simplicity, and avoiding dependency hell. All three are partially true and mostly misleading.
Flexibility is real but overrated.
You genuinely can do anything with a custom script. But “can do” and “should maintain” are different things. Every custom pagination handler, every hand-rolled retry loop, every bespoke schema migration script is code that only you understand. When you leave, and you will, the next engineer gets to reverse-engineer your cleverness.
Simplicity is a mirage.
Your first script is simple. Your third integration, with auth tokens expiring mid-run and nested JSON structures that keep shifting, is not. The challenges SQL-first teams face when adopting Python-based tools like DLT aren’t unique to SQL teams, they’re the universal tax of building data infrastructure in a general-purpose language.
Dependency hell is a valid concern, but it’s solvable.
The trade-offs between code reuse and duplication in custom Python pipelines come down to whether you’re duplicating logic across scripts or extracting a shared library. Either way, you’re maintaining something. dlt gives you a maintained, tested foundation for the 80% of ingestion that’s identical across every source.
The real cost is maintenance.
The real cost of custom pipelines isn’t the initial development, it’s the ongoing maintenance. Every API change, every schema drift, every new source that follows a slightly different pattern becomes your problem. As one data engineer put it, the team has “hell on earth pipelines” running through dlt without issues, including “ridiculous type conversions between disparate ancient data sources.” That’s the kind of problem you don’t want to own.
Schema Drift: The Silent Pipeline Killer
Here’s the thing nobody mentions when they compare dlt to custom scripts: schema drift will consume your life either way. The difference is how you handle it.
With a custom script
Schema drift means your job fails. Maybe you catch it in staging, maybe your INSERT statements break in production, maybe the data loads but with columns in the wrong order. Each failure mode requires investigation, debugging, and a fix that gets deployed through your release pipeline.
With dlt
Schema evolution is automatic. New fields get added, new tables get created, and the pipeline keeps running. Is that always what you want? Not necessarily, silent schema evolution can mask upstream changes you’d rather know about. But you can configure dlt to be strict or permissive, which is something a hand-rolled parser can’t do without substantial work.
This matters more than you’d think. The evolution of modern data stack tools shows a clear pattern: schema management is being pushed out of the ETL layer entirely. Tools like dbt handle transformation, warehouses handle storage, and ingestion tools are expected to deal with schema changes without human intervention. dlt fits that model, custom scripts don’t, unless you build it yourself.
The Performance Question Nobody’s Asking
For low-volume ingestion, performance is mostly a non-issue. You’re loading a few thousand rows, not streaming terabytes. But there’s a subtler performance consideration: developer time.
The performance implications of Python in data tooling are well-documented, SQLGlot needed a mypyc compilation just to get a 5x speedup that still doesn’t match C. But for ingestion, the bottleneck is almost never CPU. It’s the time your engineers spend writing and maintaining pipelines.
Every hour spent debugging a pagination loop is an hour not spent on transformation logic, data quality, or actual analysis. DLT’s abstraction doesn’t make your pipelines faster, it makes your team faster.
When Custom Scripts Win (We’re Not Allergic to Nuance Here)
Let me be clear: dlt isn’t always the right answer.
- Single-source, stable API: If you’re ingesting from one endpoint that hasn’t changed in years, a 50-line script is genuinely fine. You’re not paying a maintenance tax because there’s nothing to maintain.
- Pathological API designs: The one-record-at-a-time endpoint scenario we discussed earlier is a real thing. If your source forces you into async fan-out patterns, dlt’s abstraction gets in the way.
- You need CDC from Postgres: Until dlt natively supports replication slots, you’re building custom infrastructure regardless.
- Tightly coupled transformations: If your ingestion needs to transform data in-flight (not just load), you might be better off with a different tool. dlt is a loader, not a transformer.
The risks of replacing structured ETL with fragile, low-code alternatives apply here too. dlt is code-first and declarative, which puts it on the right side of the line, but it’s still an abstraction. Understanding what’s under the hood matters.
The Verdict: It’s Not Either/Or
The engineers who love dlt aren’t using it exclusively. They’re using it for the 80% of pipelines where it makes sense, and writing custom code for the 20% that don’t fit. The Python integration with enterprise databases is getting better precisely because tools like dlt are pushing the ecosystem forward.
For the use case in the original question, low volume, multiple sources, full loads, dlt is probably the right call. It’s open source, so trying it costs you nothing but an afternoon. The built-in pagination, auth handling, and schema evolution will save you weeks over the lifetime of your pipelines.
But don’t throw away your Python fundamentals. The day will come when dlt can’t handle something, an exotic auth flow, a bizarre API response shape, a source that requires custom pre-processing. When that day comes, you need to understand what’s happening under the abstraction. The engineers successfully running “hell on earth pipelines” through dlt didn’t get there by knowing dlt, they got there by knowing Python, HTTP, and data modeling well enough to make dlt work for them.
The crossroads isn’t really about dlt versus custom scripts. It’s about knowing when to stand on the shoulders of open-source tooling versus when to build your own foundation. Pick your battles wisely, and for most low-volume ingestion, the battle is already won.
