Somewhere right now, a junior data engineer is staring at a JIRA API response and wondering if they’ve already screwed up their entire career by not implementing SCD2 on every single endpoint. The confusion is understandable, the data engineering community has spent years preaching the gospel of slowly changing dimensions, and now a fresh graduate is asking the heretical question: does it even make sense to use SCD2 here?
The short answer: sometimes yes, often no, and the fact that you’re asking means you’re already ahead of most senior engineers who apply SCD2 like a default condiment without tasting the dish first.
The Bronze-Silver Pipeline That Started This Debate
A recent discussion on r/dataengineering captured the exact scenario that’s playing out in thousands of organizations. A new grad needs to ingest JIRA data, issues, projects, users, permissions, and has landed on a reasonable architecture: raw payloads land in bronze, silver gets derived using SCD2 patterns. But then comes the nagging doubt.
What happens when an API fails mid-pagination? Silver ends up with invalid data. And more fundamentally, should a permissions table have type-2 dimension history anyway?
These aren’t trivial questions. They’re the difference between a data platform that serves the business and one that exists to impress nobody but itself.
The Real Enemy: Phantom Deletes, Not Bad Rows
Here’s what most tutorials won’t tell you about API ingestion: the biggest risk isn’t corrupted data, it’s incomplete data that looks complete. This insight came through clearly in the Reddit discussion, where an experienced engineer pointed out the real danger.
A pull that dies at page 40 of 50 pages creates a catastrophic illusion. Every issue you didn’t get looks like it was deleted. If you’re running SCD2, those phantom deletions close out current records, and your history starts lying to you. The damage compounds silently, dashboards show issues disappearing, stakeholders start asking uncomfortable questions, and you’re left debugging a pipeline that technically ran successfully.
The mitigation is straightforward: count what you got against the API’s total before merging anything. JIRA tells you the total number of issues matching your query. Verify pagination completed. Compare the sum of page results to the API’s declared total. Only then is the batch trustworthy enough to merge into silver.
What SCD2 Actually Protects
Let’s get precise about what SCD2 gives you. A proper type-2 dimension tracks every state change so you can answer “what did this look like at any point in time?”
employee_id | department | effective_from | effective_to | is_current
------------|------------|----------------|--------------|------------
101 | HR | Jan 1 | Jan 10 | false
101 | IT | Jan 10 | Jan 15 | false
101 | Finance | Jan 15 | NULL | true
For JIRA data, there are objects where this genuinely matters. Issue status changes, assignee changes, and priority changes represent meaningful history. If a stakeholder asks “when did this incident shift from sev-2 to sev-1?”, an SCD2 pattern on the issue dimension gives you the answer instantly. Project settings and permissions also benefit from change tracking, knowing who had access when is frequently an audit requirement, not just a nice-to-have.
But the key phrase is selective application. The community consensus, echoed across the discussion thread, is that SCD2 makes sense for objects where history matters and falls apart as a blanket strategy. Users, for example, rarely need dimension history. If someone changes their display name, do you need to know what their avatar looked like six months ago? Almost certainly not.
The Idempotency Question
One commenter raised the right counterpoint: if an API fails, shouldn’t an SCD2-friendly date strategy automatically handle it? The idea would be to capture an API success watermark date, ensuring silver only updates on net new dated rows.
This works, but only if you’ve built idempotency into your pipeline from the start. The pattern requires:
- A run ID or timestamp assigned to each extraction attempt
- Bronze loading that’s resilient to retries (no duplicate artifact accumulation)
- A success marker only written after complete pagination
- Silver merges that reference the last successful watermark
If page 37 of 50 fails, you keep the previous silver state and retry the entire bronze load. The incremental logic never sees partial data because the watermark doesn’t advance on failure.
This is the pattern that separates production-grade pipelines from weekend projects. Implementing it isn’t glamorous work, but it prevents the exact failure mode that terrifies fresh graduates and seasoned pros alike.
The Overhead Stack: When SCD2 Becomes a Tax
Here’s the uncomfortable reality: SCD2 isn’t free. Every type-2 dimension adds operational weight to your pipeline:
- Merge complexity: Handling multiple records for the same key arriving in the same batch. Is it a duplicate or a legitimate sequence of changes? Deduplication via window functions or processing changes in event order, pick one and document it.
- Validation burden: You need guarantees that exactly one current record exists per business key. The SQL check is simple (
GROUP BY employee_id HAVING COUNT(*) > 1), but the maintenance isn’t. - Read complexity: Consumers need
effective_from/effective_tofilters oris_current = truepredicates. Every analyst query gets slightly harder. - Storage amplification: Every change creates a new version. Over time, that’s a long tail of near-identical rows.
For a lightweight or early-stage pipeline, this overhead can be the difference between shipping in two weeks and shipping in two months. And if the API data feeding your SCD2 dimension has no audit requirement and no historical analysis use case, you’ve built complexity with zero payoff.
The Medallion Architecture Nuance
There’s also a question of where SCD2 should live. Raw bronze data should stay immutable, that’s non-negotiable. But silver is where the pattern gets debated.
One pragmatic approach: keep raw responses in bronze permanently, apply SCD2 only where history explicitly matters, and use simple upserts (SCD1-style) elsewhere. This borrowed from the dbt incremental strategy docs, which note that basic incremental strategies “insert selected records into the destination table without updating or deleting existing data” and don’t align directly with type 1 or type 2 SCD patterns.
Append-only tables have a place. Regulatory/compliance data, event logs, and immutable audit trails don’t need SCD2, they need append-only with timestamps. The historical record is inherent in the data itself, not superimposed by a dimension pattern.
The “No Understand, Only Apply” Problem
The deeper issue is that SCD2 became a default answer in interviews and tutorials without the accompanying nuance about when it applies. The PySpark & Databricks interview material circulating online reinforces this, treating SCD2 as a standard ETL pattern that “reduces thousands of lines of code to a few declarative statements.”
Modern declarative tools like Databricks Lakeflow pipelines do make SCD2 easier to implement. The “Auto CDC API” now handles SCD Type 1 and Type 2 change data capture events, including out-of-order events, without manual watermarking code. When the plumbing is this abstracted, the temptation to apply it everywhere grows stronger.
But easier implementation doesn’t make it the right choice everywhere. It just makes the wrong choice cheaper to execute.
A Decision Framework That Actually Works
Instead of asking “should I use SCD2 for API ingestion?”, reframe the question around consumption:
Apply SCD2 when:
– You need point-in-time analysis (“what was the state on March 15?”
– Audit requirements demand change history
– The API data feeds dimensions that change slowly and matter to reporting
– The cost of reconstructing history from raw bronze is higher than maintaining SCD2
Skip SCD2 when:
– You’re in exploration mode and don’t know what queries will matter
– The data is high-volume transactional data (facts, not dimensions)
– History exists only as “nice to have”, not as a requirement
– Your team is small and moving fast, simplicity wins
Use something else entirely when:
– You just need current state → SCD1 upserts
– You need immutable records → append-only with timestamps
– You need both current and historical state → SCD2, but only here
For raw API ingestion specifically, there’s a strong argument that bronze should be raw JSON/Parquet snapshots with metadata columns (extraction timestamp, API endpoint, run ID). Silver gets selected SCD2 treatment on justifiable dimensions. Gold remains aggregation purpose-built.
The Tools Should Serve You
Maybe the most useful shift is toward purpose-built ingestion tools that handle the messy parts, pagination, retries, failure recovery, so you can focus on modeling decisions like SCD2. There’s a genuine trade-off between custom and purpose-built tools for API data ingestion, and another on the balance between code reuse and pipeline simplicity. These aren’t abstract debates, they determine whether your team spends cycles building pagination wrappers or answering business questions with historical data.
The same logic applies to the broader ecosystem. Advanced table formats like Iceberg handle versioning at the file level, giving you time travel without dimension modeling. And the AI data ingestion gold rush is producing tools that automate entire ingestion pipelines, but automated complexity isn’t automatically good complexity.
History When It Matters, Simplicity When It Doesn’t
The debate over SCD2 in API ingestion isn’t really about SCD2. It’s about whether we’re building data platforms that serve the business or platforms that demonstrate our mastery of data engineering patterns. The best engineers apply the lightest tool that solves the problem. The rest apply every tool they know to every problem they see.
Fresh-out-of-university engineers asking “does SCD2 even make sense here?” are asking the exact question their senior counterparts should be asking more often. So no, it’s not overkill to question the pattern. It’s overkill to apply it without question.
Build bronze raw, test your pagination integrity, and only put SCD2 where history has an actual consumer. Your stakeholders will thank you. Your future self will thank you. And the data will tell the truth.
