Your Spark Pipelines Are Still Writing Code Like It’s 2015. SDP Is Here to Fix That.

Let’s be honest about what most Spark ETL looks like in the wild: a thousand-line notebook where you manually orchestrate reads, transformations, and writes, praying that the dependency order doesn’t change when someone refactors a function. You’re not building a pipeline, you’re writing a recipe for disaster.

Spark Declarative Pipelines (SDP), now a native capability in Apache Spark 4.1, is the industry’s most serious attempt to kill that pattern. And for once, the hype might be justified. The framework isn’t some bolt-on abstraction. It’s core Spark now. No external dependencies. No new learning curve. You write what you want the data to look like, and Spark handles the dependency resolution, execution order, and parallelization.

The fact that this is being taught at Devoxx and DataEngBytes, conferences with a notorious ~90% CFP rejection rate, suggests this isn’t just vendor marketing. This is the community saying “we need this, and we need it now.”

From “Imperative Spaghetti” to “Declarative Intent”

The core shift is philosophical. Traditional Spark jobs force you to micromanage the how: read this source, apply this transformation, write to this table. You also have to control the execution sequence, manage retries, and handle incremental state, all yourself. It’s procedural, brittle, and the reason your team’s onboarding docs are 40 pages long.

SDP inverts the model. You describe the outcome, the datasets you want, their sources, and their relationships, and Spark figures out the rest. It’s the difference between telling a chef exactly which knife to use and when, versus telling them “I want a medium-rare ribeye with chimichurri.”

Under the hood, SDP pipelines are defined in YAML and composed from Python and SQL source files. The framework automatically:
– Resolves dependencies between datasets and transformations
– Determines execution order across pipeline steps
– Runs independent tasks in parallel to improve performance

The building blocks are refreshingly simple:

A recent hands-on tutorial demonstrated a production-grade pipeline tracking every aircraft in the sky, millions of live IoT events streaming in every second, built in just a few lines of code using SDP. The author, who presented this at those high-bar conferences, walked through the exact same use case in two environments: 100% open source with PySpark, and on Databricks’ Lakeflow Free Edition.

What’s notable is the use case itself. It’s not another generic “read CSV, write Parquet” demo. It pulls live data from the OpenSky Network, a crowdsourced air traffic surveillance system fed by thousands of volunteers with low-cost ADS-B receivers. Every pipeline run pulls real-time position, velocity, and altitude updates from aircraft currently in flight. This is production-scale IoT data, not a static sample file.

The OpenSky Data Source: A Gift to the Community

A key enabler of that demo is the custom PySpark data source built for the OpenSky REST API. It’s open source, works in plain Spark just as well as in SDP, and anyone can use it. AWS flight tracking data isn’t typically something you’d associate with a “getting started” tutorial, but it makes the abstraction immediately tangible.

This is the kind of thing that moves the needle on adoption, real, relatable, and free. The data exists because aviation enthusiasts around the world set up receivers (and you can feed your data to the OpenSky Network to become one of them).

Lakeflow: The Enterprise On-Ramp

Let’s address the elephant in the room: SDP was born as Databricks’ Delta Live Tables (DLT). Databricks contributed the core framework to Apache Spark at Data + AI Summit 2025, and DLT has since evolved into Lakeflow Declarative Pipelines, the managed, enterprise version.

The relationship is symbiotic. Lakeflow runs 100% of the open source SDP core, so your pipeline code is identical in both environments. But Databricks adds platform capabilities that production teams need but shouldn’t have to build themselves:

• Serverless compute with automatic scaling and significant performance improvements
• A built-in pipeline editor with AI-powered data exploration
• ETL pipeline generation with Genie Code
• Deeper Unity Catalog integration, including automatic lineage tracking

For a team that wants to avoid the operational overhead of managing Spark clusters, Lakeflow is the fastest path to a running pipeline. The Free Edition requires zero local setup and no credit card, everything runs in the cloud.

The strategy is smart: by making the core open source, Databricks positions itself as the best managed option rather than the only option. If you’re running plain Spark on your laptop or an EMR cluster, you can use SDP today with no vendor lock-in. If you want it fully managed with governance and lineage, Lakeflow is there.

This is a significant departure from the days of proprietary orchestration tools. The pattern mirrors what we saw with Kubernetes, the open standard becomes the floor, and the commercial offering competes on experience and integration, not on being the only game in town. It also connects to the broader evolution of open, vendor-neutral data standards, which is a theme we’ve explored before.

The SQL Gold Layer Problem: Solved Elegantly

One of the most promising developments is how SDP handles the “Gold layer” of a medallion architecture. A feature request in the SDP-META repo outlines an approach where Bronze and Silver remain metadata-driven, but the Gold layer, where customer-specific business logic lives, is expressed as native SDP SQL.

The architecture is refreshingly clean:

Onboarding YAML/JSON
        ↓
Bronze/Silver DataflowSpec
        ↓
Bronze pipeline
        ↓
Silver pipeline
        ↓
Unity Catalog Silver tables
        ↓
Native SDP SQL Gold pipeline
        ↓
Gold materialized views

Instead of wrestling with yet another metadata-driven abstraction for business logic, you write plain SQL:

-- Silver inputs use configuration-driven, fully qualified names
CREATE OR REFRESH MATERIALIZED VIEW customer_360 AS
SELECT *
FROM ${silver_catalog}.${silver_schema}.customers;

-- Gold-to-Gold references use unqualified names
CREATE OR REFRESH MATERIALIZED VIEW high_value_customers AS
SELECT *
FROM customer_360;

SDP determines Gold execution order from the SQL dependency graph, no file ordering, no manual orchestration. The framework’s validation layer is similarly thoughtful, rejecting deprecated LIVE TABLE syntax and ensuring each SQL file declares exactly one materialized view or streaming table.

This is a much-needed sanity check for the data engineering community, which has seen more than a few frameworks promise automation and deliver complexity. The trade-offs between Spark and modern ELT tools like dbt have been well-documented, and SDP’s SQL-first approach for Gold layers feels like the best of both worlds, the compute power of Spark with the developer experience of a lean SQL transformation tool.

Where Does This Fit in the Big Data Ecosystem?

SDP’s rise isn’t happening in a vacuum. It’s part of a broader trend toward declarative, intent-based systems, the same shift we’ve seen in everything from infrastructure-as-code to the debates around SQL versus the broader Spark ecosystem. The SQL vs. Spark ecosystem discussion is being reheated by SDP’s SQL-native capabilities.

There’s also the matter of platforms. The Azure lakehouse market, for instance, is bifurcating between engineer-controlled compute (Databricks) and integrated SaaS (Microsoft Fabric). Both write the same Delta Parquet on ADLS Gen2, but they differ dramatically in who operates the compute. SDP is central to the Databricks story, it’s how you write ETL once and run it anywhere Spark runs.

For teams weighing their options, the decision often comes down to workload shape. Spiky, idle-heavy batch workloads benefit from Databricks’ elastic DBU pricing. Steady, BI-anchored, mixed-skill teams lean toward the predictability of a fixed Fabric capacity. SDP doesn’t resolve that debate, it thrives on either side of it, as long as Spark is underneath.

The $64,000 Question: Is This Actually Simpler?

Skeptics will argue that SDP is just another abstraction layer. You’re trading imperative code for a framework that has its own YAML structure, its own lifecycle, and its own quirks. That’s a fair criticism, moving the complexity somewhere else doesn’t eliminate it.

But there’s a crucial difference. The complexity SDP removes is the kind that grows non-linearly with pipeline size: dependency management, state handling, and the subtle race conditions that emerge when independent transformations inadvertently depend on each other. These aren’t problems you can solve with better code quality. They’re structural.

SDP’s automatic dependency resolution and parallel execution eliminate an entire class of bugs. When your pipeline grows from 10 tables to 100, the framework scales with you. The same can’t be said for a notebook where someone hardcoded a time.sleep(60) between steps because “it wasn’t working otherwise.”

The OpenSky demo makes this tangible. What used to require serious engineering effort, ingesting live IoT data, transforming it in a streaming context, and writing it to tables, is accomplished in a coffee break with a few lines of code. And the requirements are modest: Python 3.12, Java 17, and PySpark 4.1.

The Takeaway for Data Teams

If you’re a data engineer, SDP should be on your radar. Not because it’s the latest shiny object, but because it represents a genuine shift in how Spark-based pipelines are authored and maintained. The fact that it’s native to Apache Spark 4.1 means it’s not going away.

Start small. Clone the SDP OpenSky tutorial repo, run it locally with open-source PySpark, and see for yourself how quickly you can go from zero to a working streaming pipeline. Then consider the Lakeflow tutorial if you want to see the managed experience.

Your future self, the one who no longer has to trace dependencies through a 2,000-line notebook, will thank you. And if you’re wondering whether this means a fundamental shift in how we build data infrastructure, the answer is yes. The question is whether you’ll be leading the change or documenting it for the next person.

For more on how the data engineering landscape is evolving, check out our breakdown of the Rust wave and whether we’re finally moving beyond Spark and Java. Spoiler: Spark isn’t going anywhere yet, but the way we use it is changing fast.