Michael Tall — Header
Abstract Delta table structure on engineering monitor

Delta Lake in Fabric: V order, compaction and runtimes for Engineers

Delta Lake in Microsoft Fabric is the platform’s default, versioned table format, stored as Delta Parquet, that gives you ACID transactions, time travel and single-copy sharing across OneLake. Every workload in Fabric, from lakehouses to warehouses to KQL databases, reads and writes this same format by default. That matters because it removes the need to copy data between engines just to run different workloads against it.

TL;DR:

  • Delta Lake in Fabric provides ACID guarantees, time travel, and a single copy of data that supports multiple workloads without duplication.
  • Delta tables include Parquet data files, a transaction log, and optional checkpoints; automatic maintenance optimizes read and write performance.
  • Fabric’s unified Delta format enables direct, real-time access for Spark, SQL, Power BI, and data pipelines, reducing data movement and synchronization issues.
  • Support varies across Fabric experiences, with full feature support in PySpark and limited capabilities in Python notebooks and some connectors.
  • Routine maintenance like vacuuming and careful protocol upgrades are essential to ensure data integrity, concurrency, and future compatibility.
Datataal
Build More Reliable Fabric Architectures
Datataal designs scalable Microsoft Fabric data architectures that simplify analytics, strengthen governance, and support real-time business insights.

Explore Datataal’s Fabric expertise

Table of Contents

What Delta Lake tables actually are

A Delta table is not a new file format. It is Parquet data files paired with a transaction log, stored in a folder called _delta_log, plus periodic checkpoints that summarise that log. The log records every insert, update, delete and schema change as a sequence of committed actions, which is what gives Delta its ACID guarantees.

Plain Parquet has no equivalent record. It stores rows efficiently but has no concept of a transaction, so two writers touching the same files can corrupt each other’s work, and there is no way to roll back a bad write. Delta Lake extends Parquet with that versioned log, which is what makes atomic commits, rollbacks and reliable multi-writer behaviour possible.

In practice, this gives you:

  • Atomic commits, so a failed write never leaves a table half-updated.
  • Schema enforcement, which rejects incompatible writes before they land.
  • Time travel, letting you query or restore an earlier table version.
  • Change data feed, which surfaces row-level inserts, updates and deletes for downstream processing.

Delta table structure inside OneLake

When you open a lakehouse in Fabric, what you are actually browsing is this folder structure. Each Delta table has a root folder containing the data files and a _delta_log subfolder holding JSON commit files and checkpoint files.

  • Data files: Parquet files holding the actual rows, often many small files before compaction runs.
  • _delta_log: an ordered sequence of JSON files, one per commit, describing added and removed files, schema changes and metadata.
  • Checkpoints: periodic Parquet summaries of the log that let engines skip replaying every JSON commit from the start.

Checkpoint format matters for read speed: older V1 checkpoints store the full log state in a single file, while newer V2 checkpoints split this into sidecar files that large tables can read more efficiently. Fabric’s lakehouse explorer reads this structure directly to list tables and columns. Shortcuts work the same way: a shortcut is a pointer to a Delta table living elsewhere (another lakehouse, or external storage), so Fabric reads the same _delta_log without copying the underlying files.

How Fabric uses Delta Lake across workloads

OneLake stores Delta Parquet as the default format, which means one physical copy of a table can serve a Spark notebook, a SQL query and a Power BI report without separate exports. This single-copy model is the practical payoff of Delta being Fabric’s default table format: fewer duplicated pipelines, less drift between BI and engineering datasets.

Direct Lake is the clearest example of that benefit. Rather than importing data into a semantic model, Power BI reads Delta tables in OneLake directly, which removes the import-refresh cycle and brings reporting closer to real time. Our own breakdown of Direct Lake trade-offs covers when this beats a traditional import model.

Several Fabric experiences read and write this same format:

  1. Spark runtimes, for full read and write support including advanced table features.
  2. Data pipelines, for orchestrated loads into Delta tables at scale.
  3. KQL databases, which can expose Delta-formatted data for analytical queries.
  4. SQL endpoints, which surface lakehouse tables for T-SQL querying without moving data.

Performance and optimisation in Fabric

Fabric applies V-order automatically when Spark writes Parquet files, a write-time optimisation that improves downstream read performance across Fabric engines. It also runs automatic table maintenance including compaction, which merges small files produced by frequent writes into fewer, larger ones.

That covers a lot of routine tuning without any action from you. What it does not remove is the need to think about partitioning: partitioning by a high-cardinality column you rarely filter on will hurt performance rather than help it, regardless of how well the files are compacted. Manual OPTIMIZE runs still earn their keep on tables with heavy streaming writes or where automatic compaction hasn’t caught up with ingestion volume.

Pro Tip: Before running a manual OPTIMIZE job, check whether automatic compaction has already handled the table. Blanket optimisation on tables Fabric is already maintaining wastes compute for little gain.

Creating, loading and querying Delta tables

You can create Delta tables in Fabric several ways, and the runtime you pick affects which features you get.

  • DataFrameWriter in a Spark notebook, writing directly to a lakehouse path.
  • SQL CREATE TABLE statements against the lakehouse or warehouse SQL endpoint.
  • Data pipelines, orchestrating loads from external sources into Delta targets.
  • The lakehouse UI, for quick table creation from uploaded files.

For full feature support, use PySpark notebooks. Python notebooks rely on the deltars/deltalake libraries rather than the Spark Delta connector, and can lack support for deletion vectors, column mapping writes and liquid clustering. If a Python notebook fails to read a table because of one of these features, switching that step to a PySpark notebook is usually the fix, and it’s worth building that choice into your pipeline design rather than discovering it in production.

Once tables exist, querying is straightforward: Spark SQL and PySpark for engineering workloads, SQL endpoints for analysts, and Direct Lake for Power BI reports that need near real-time figures without a refresh schedule. Import mode still has a place for heavily transformed or aggregated semantic models where Direct Lake’s direct read isn’t the better fit.

Compatibility across Fabric experiences

Delta Lake has moved from a small set of protocol version numbers to a Table Features model, where individual capabilities like deletion vectors or column mapping are enabled independently. This matters practically: enabling a table feature can make a table unreadable by older or less capable clients, even though the table itself is perfectly valid.

Support varies by Fabric experience:

  • PySpark notebooks support the full current feature set.
  • Python notebooks and some pipeline connectors support a narrower set, and may fail on deletion vectors or column mapping.
  • KQL and SQL endpoints read Delta tables but don’t write advanced features themselves.

For sharing Delta tables outside Fabric, Universal Format (UniForm) generates Iceberg or Hudi metadata asynchronously alongside the Delta log, so external Iceberg or Hudi readers can query the same files without a separate copy. It requires column mapping and a minimum protocol version, and it does not support writing back in those formats, only reading.

Maintenance, concurrency and safe upgrades

Routine maintenance is mostly about vacuum and checkpoint frequency. VACUUM removes files no longer referenced by the current table version, and setting too short a retention window can break time travel or in-flight readers, so keep retention generous unless storage cost forces the issue.

Concurrent writes are handled through optimistic concurrency control: Delta stages new files, validates against the latest log, and commits only if nothing conflicts, failing with a retryable exception otherwise rather than corrupting the table. This is why retry logic in pipelines matters more than trying to avoid all concurrent writes.

Delta concurrent writes and retry flow

Before enabling a new table feature or upgrading protocol versions, test in a staging environment first and confirm every consumer, Power BI, Spark jobs and any external reader, can still open the table. Rolling an upgrade straight to production is the most common way teams accidentally lock themselves out of their own data.

What I’d tell a team starting this today

Most teams overestimate how much manual Delta tuning they need in Fabric and underestimate how much thought their table feature and runtime choices deserve. The platform genuinely automates V-order and compaction well. Where I see projects run into trouble is picking Python notebooks for convenience early on, then hitting a wall when deletion vectors appear in a table a PySpark job wrote.

A short checklist helps: convert raw landing data to Delta as early as the bronze layer, keep gold tables as managed tables feeding Direct Lake, and pilot any protocol upgrade against a staging copy before touching production. Our note on OneLake and duplication covers how this fits into a wider medallion layout.

— Michael

Get Delta Lake and Fabric working for you

Knowing how Delta tables, V-order and Direct Lake fit together is one thing. Getting a Fabric estate built on that foundation, with proper medallion layering, governance and reporting that doesn’t fall over the first time a protocol upgrade goes wrong, is another. I work directly with your team through the full engagement, no junior staff or handoffs in between, delivering the architecture and the knowledge transfer so your people can maintain it afterwards.

If your organisation is planning a move onto Fabric or already has Delta tables that need proper structure, my Microsoft Fabric consulting services cover architecture, data engineering and Power BI delivery from a single point of contact, starting from $20 per hour. Book a consultation and we can look at where your current setup stands against what Fabric can actually do for it.

Sources

FAQ

What is Delta Lake and why is it used?

Delta Lake is an open table format that adds a transaction log on top of Parquet files, giving ACID transactions, schema enforcement and time travel. It is used because it lets multiple engines and pipelines write and read the same tables reliably without corrupting data or requiring separate copies for each workload.

What is the difference between Parquet and Delta?

Parquet is a columnar file format with no built-in transaction tracking, so it cannot guarantee atomic writes or support rollbacks on its own. Delta Lake wraps Parquet files with a transaction log, which adds versioning, ACID guarantees and the ability to time travel to an earlier table state.

What’s the difference between Delta Lake and Iceberg?

Both are open table formats built for large-scale analytics with ACID transactions, but they use different metadata structures and originate from different ecosystems. Delta’s Universal Format (UniForm) can generate Iceberg-compatible metadata asynchronously so Iceberg readers can query Delta tables, though it requires column mapping and a minimum protocol version, and it does not support writing in Iceberg format.

What is Delta Parquet format in Fabric?

Delta Parquet is simply the combination Fabric uses by default: standard Parquet data files paired with the _delta_log transaction log. OneLake stores this as the default format across lakehouses, warehouses and KQL databases, so a single physical table can serve multiple Fabric workloads without duplication.

Microsoft Fabric blog