Michael Tall — Header
Engineer configuring incremental refresh settings

Engineers: 12 Steps to Configure Incremental Refresh in Fabric

Incremental refresh in Dataflow Gen2 updates only the time-based buckets that changed, replacing just those buckets rather than reprocessing an entire table. The result is faster refreshes and lower compute cost, but only when three conditions hold: your workspace sits on a Fabric capacity, your destination supports incremental refresh, and your query folds correctly against the source.

TL;DR:

  • Incremental refresh requires a Fabric capacity, a Date or DateTime source column, query folding, and a destination that supports native replace operations.
  • Supported destinations include Fabric Lakehouse, Fabric Warehouse, and Azure SQL Database; unsupported destinations need staging tables and separate replace queries.
  • Proper configuration involves verifying query folding, setting correct bucket sizes aligned with data change frequency, and using a reliable change-detection column to prevent missed late data.
  • Largest performance gains come from small buckets, accurate filtering, and monitoring refresh durations to ensure source updates match the bucket size.
  • Failures often stem from schema drift, unsupported destination types, or broken query folding, which cause silent full refreshes or duplicated records.
Datataal
datataal.com
Build A More Reliable Fabric Foundation
Datataal designs scalable Microsoft Fabric architectures that simplify reporting, strengthen governance, and support real-time business insights.

Explore Datataal

Table of Contents

Prerequisites for incremental refresh in Fabric

Before you touch the incremental refresh dialog, check the fundamentals. Skipping this step is the most common reason engineers see timeouts or silent full refreshes later.

According to Microsoft’s Dataflow Gen2 documentation, incremental refresh requires a Fabric capacity, a source column typed as Date, DateTime or DateTimeZone, and a destination that explicitly supports the feature.

  • Confirm your workspace runs on a Fabric capacity with the appropriate licensing tier.
  • Verify the source exposes a genuine Date, DateTime or DateTimeZone column you can filter on.
  • Check that your query folds against the source; non-folding queries defeat the purpose of bucketed refresh.
  • Confirm the destination supports incremental refresh natively, or plan a staging table if it does not.
  • Note that Fabric enforces a maximum number of buckets, so extremely fine-grained bucketing on long histories can hit that ceiling.

If your destination has a fixed schema, make sure the query output matches it exactly before you enable incremental refresh. Schema drift after the first publish is a frequent cause of failed subsequent runs.

Which destinations support incremental refresh in Dataflow Gen2

Not every Fabric destination behaves the same way once incremental refresh is switched on. The Microsoft Learn documentation lists which destinations support the feature and confirms that the only supported update method is replace, meaning Fabric always overwrites the contents of an affected bucket rather than merging or appending into it.

  • Fabric Lakehouse: supported, and the most common target for incremental Dataflow Gen2 patterns.
  • Fabric Warehouse: supported, useful when downstream consumers expect a relational surface.
  • Azure SQL Database: supported as an external destination.
  • Destinations without native support: not usable directly. You need a staging pattern instead.

When your final destination lacks native support, the practical answer is a two-query pattern: one query lands incrementally into a supported staging destination, and a second query performs a controlled replace or merge into the true destination. Microsoft’s own pattern for incrementally amassing data with Dataflow Gen2 describes this approach as a way to avoid exposing a partially written table to downstream reports while the replace operation runs. Treat replace-only semantics as a design constraint, not a limitation to work around: it is what makes the bucket model safe to run unattended.

Step-by-step: configuring incremental refresh in Dataflow Gen2

Once your prerequisites are confirmed, configuration itself is a short, repeatable sequence in the Dataflow Gen2 editor.

  1. Open or create your Dataflow Gen2 query and connect it to the source table you want to load incrementally.
  2. Confirm the date or timestamp column you will filter on is present in the output and correctly typed, not a text column that merely looks like a date.
  3. Check query folding by reviewing the step-by-step query plan; a query that stops folding partway through will pull far more data than intended.
  4. If folding breaks, simplify the offending step or push the transformation downstream, after the incremental filter has already been applied at the source.
  5. Right-click the query in the queries pane and select Incremental refresh from the context menu, as described in the Dataflow Gen2 documentation.
  6. Set the filtering column to the date or timestamp field you verified earlier.
  7. Define the extraction period, the historical window Fabric loads on first run, for example the last two years of records.
  8. Choose a bucket size that matches how often your source data actually changes; daily buckets suit daily-refreshed sources, monthly buckets suit slower-moving tables.
  9. Optionally set a separate change-detection column if you want Fabric to compare a modified timestamp rather than relying solely on the partitioning column.
  10. Enable the require fold option if you want the dataflow to fail loudly rather than silently pull an entire table when folding breaks.
  11. Publish the dataflow and let the initial full load run; this populates every bucket for the first time.
  12. Schedule the dataflow, or trigger it from a Data Factory pipeline, so subsequent runs evaluate and refresh only the buckets that changed.

If your final destination cannot accept replace operations directly, add a staging query that lands data incrementally, then a second query that performs the full replace or merge into the production table on a schedule that runs immediately after the staging load completes.

Pro Tip: Run the initial full load during a low-traffic window and watch its duration closely: it is the best early indicator of whether your bucket size and extraction period are realistic for production.

How Dataflow Gen2 decides what to refresh

The mechanics behind incremental refresh are straightforward once you see the four-step cycle Microsoft describes in its Fabric blog on the public preview: publish and automate, evaluate changes, retrieve data, and replace data.

Four-step incremental refresh cycle

On each scheduled run, it evaluates whether the maximum value of the change-detection column (or the filtering column itself, when no separate column is set) has moved since the previous refresh for each bucket. Buckets with no change are left untouched entirely, which is where most of the performance gain comes from.

Buckets flagged as changed are retrieved in parallel from the source into staging, then written back to the destination using the replace method, meaning the entire bucket’s prior contents are overwritten with the newly retrieved set. Historical data sitting in buckets that were not flagged remains exactly as it was, untouched by the run.

This has a practical implication for late-arriving rows: if a record lands in the source with an old timestamp after its bucket has already been marked unchanged and skipped, it will not appear in the destination until that bucket is next evaluated as changed or a manual refresh forces it. Choosing a change-detection column that reliably reflects when a row was last touched, not just when the business event occurred, reduces this risk considerably.

Understanding the incremental refresh settings

Each field in the incremental refresh dialog controls a specific behaviour, and getting them right matters more than getting the feature merely switched on.

  • Filtering column: usually a business-event date, but pairing it with a separate change-detection column catches corrections made to historical rows, which a pure event date would miss.
  • Extraction period: how far back the initial load reaches. Set it to cover your genuine reporting history, not an arbitrary round number.
  • Bucket size: smaller buckets increase parallelism and shrink the blast radius of each replace, while larger buckets reduce the management overhead of tracking many partitions but replace more rows per run.
  • Change-detection column: an optional field, often a last-modified timestamp, that Fabric checks against its previous maximum value to decide whether a bucket needs refreshing.
  • Require fold: forces the dataflow to fail rather than fall back to pulling full data locally when the query cannot fold the incremental filter.

Bucket size and folding are the two settings that determine most of your refresh cost. According to Microsoft’s incremental refresh documentation, bucket sizing should align with source update cadence and late-arrival behaviour, and refresh duration per bucket should be monitored as that cadence changes.

Enable “only extract new data when the max changes” style behaviour by relying on the change-detection column: it stops Fabric reprocessing a bucket where only unrelated rows moved.

Tuning and best practices for reliable refreshes

Getting incremental refresh into production is one thing; keeping it reliable as data volumes grow is another.

  • Validate query folding before you enable incremental refresh, and push filters as close to the source as possible.
  • Test with a small extraction window first, then widen it once you have confirmed correct bucket behaviour.
  • Align bucket size with your source’s actual update cadence rather than an arbitrary default, and monitor per-bucket refresh duration over time.
  • For very large historical backfills, bootstrap the initial load separately using SSMS, notebooks or pipelines rather than forcing one enormous first run through the dataflow.
  • If your source data is mutable rather than append-only, check whether materialised lake views and Delta change data feed suit your case better, since CDF-based incremental processing handles updates and deletes more explicitly than bucket replacement does.

Pro Tip: Keep a simple log of bucket refresh durations from your pipeline runs; a sudden increase almost always means your bucket size no longer matches how fast the source is growing.

Troubleshooting common incremental refresh problems

Most incremental refresh issues trace back to one of four causes.

  1. Initial refresh timeouts: bootstrap the first load in smaller historical chunks rather than one pass across the full extraction period.
  2. Silent folding failures: review the query plan step by step; a step that breaks folding forces Fabric to pull data locally, so simplify or relocate that transform, and consider enabling require fold to catch this earlier.
  3. Duplicate values after refresh: usually a sign the filtering column is not a reliable partition key. Refresh the affected bucket in isolation to recover, then correct the column choice.
  4. Unexpected full refresh: this happens when the destination doesn’t support replace semantics correctly, when unsupported SQL constructs are used, or when updates and deletes occur without change data feed configured, as the materialised lake views guidance notes for CDF-dependent scenarios.

Practitioner notes from real Fabric implementations

A few patterns consistently separate reliable incremental refresh setups from fragile ones. Use a dedicated change-detection column, distinct from your business-event date, whenever the source updates historical rows after the fact; a pure event date will miss those corrections entirely. When the destination cannot accept replace operations natively, the staging-and-replace pattern protects production tables from partial writes. Choose materialised lake views with change data feed over Dataflow Gen2 buckets when your source is genuinely mutable rather than append-only. Before launch, confirm capacity, folding, destination support and bucket sizing, in that order.

Practitioner notes from real Fabric implementations — overview diagram

When incremental refresh is the right choice

Incremental refresh earns its keep on large, append-oriented tables with a reliable time-based watermark, transaction logs, event streams, daily extracts. It struggles when data is highly mutable or when your query depends on SQL constructs that never fold, since both push Fabric back towards full refreshes regardless of configuration. Verify destination compatibility and folding first; everything else is tuning.

— Michael

Getting incremental refresh right without the trial and error

Most teams lose time not on the incremental refresh dialog itself but on the folding checks, bucket sizing and staging design around it. I work directly with organisations to assess their Dataflow Gen2 setup, implement incremental refresh against the correct destination pattern, bootstrap the initial load, and hand over a monitored, documented configuration your own team can run.

Datataal

That engagement moves through three stages: an assessment of your current sources and destinations, hands-on implementation and tuning, and a knowledge transfer session so your team owns the result. You can see the scope of this work on the Microsoft Fabric consulting services page and get in touch to discuss your setup.

Where to read more on incremental refresh in Fabric

For the technical detail beyond what fits here, these are the primary references worth bookmarking.

Sources

FAQ

What is incremental refresh?

Incremental refresh is a technique that updates only the portion of a dataset that has changed since the last run, instead of reprocessing the entire table. In Dataflow Gen2, this means Fabric evaluates time-based buckets and replaces only the ones flagged as changed, as described in Microsoft’s documentation.

What is the difference between CDC and incremental load?

Change data capture (CDC) tracks individual row-level inserts, updates and deletes as they happen, often through a change data feed, and is built to handle mutable data correctly. Incremental load in Dataflow Gen2 instead partitions data into time buckets and compares a watermark value, which works well for append-oriented data but relies on change data feed for reliable handling of updates and deletes, as the materialised lake views guidance explains.

How do I use incremental refresh in Power Query?

Incremental refresh in Power Query for Power BI semantic models uses RangeStart and RangeEnd parameters to define partitions, a different mechanism from Dataflow Gen2’s bucket settings, and the service creates the actual partitions after the first refresh completes. Query folding is the most common point of failure in both approaches, so verify it in the query editor before scheduling, per the Power BI incremental refresh documentation.

Why does Dataflow Gen2 fall back to a full refresh instead of incremental?

This typically happens when the destination does not support replace semantics correctly, the query contains SQL constructs that prevent folding, or the source involves updates and deletes without change data feed enabled. Reviewing the query plan for folding breaks and confirming destination support are the first two checks to run.

Microsoft Fabric blog