Fabric data quality, in the Microsoft context, combines Purview-driven profiling and scan workflows for Lakehouse assets, declarative constraint checks in Materialized Lake Views (MLVs), and pipeline-level validations. Your immediate next action: run a Data Map scan using a service principal, associate your Lakehouse with a data product in the Unified Catalog, then trigger a profile and your first data quality scan using the Purview MSI.
When Fabric-native checks are enough:
- Your tables are in Delta or Iceberg format and already registered in the Data Map
- You need profiling, null/range/pattern rules, and a scorecard without additional tooling spend
- Your governance requirements are met by Purview lineage, MLV constraint logs, and Power BI scorecards
When to pair with third-party tools:
- You need cross-system referential integrity checks that span sources outside OneLake
- Your pipelines already use dbt or Great Expectations and you want test results in the same CI/CD loop
- Enterprise MDM, data stewardship workflows, or complex rule libraries (Informatica IDMC, Ataccama) are required by your organisation’s data governance standards
Key takeaways
Fabric data quality requires Delta or Iceberg table formats, a completed Data Map scan, and separate credentials for Data Map versus Data Quality scans before a single rule can run reliably in production.
| Point | Details |
|---|---|
| Format is a hard prerequisite | Delta or Iceberg tables are required; convert Parquet sources before registering DQ targets. |
| Two credentials, two scan types | Use a service principal for Data Map scans and the Purview MSI for data quality scans — mixing them causes authentication failures. |
| Start narrow, not broad | Profile one high-value data product first; author three to five rules calibrated to the profiling baseline before expanding. |
| Gate at layer transitions | Place MLV constraints at each Medallion layer boundary so bad rows never reach serving models silently. |
| Pair tools to governance needs | Use Fabric-native checks for most workloads; add dbt tests or Great Expectations for CI/CD gates; adopt Informatica IDMC or Ataccama only when MDM or multi-domain stewardship is required. |
Table of Contents
- How does Microsoft Fabric implement and organise data quality?
- How do you set up Fabric Lakehouse data quality from scratch?
- How do you profile tables, create rules, and run a data quality scan?
- What are the known limitations and gotchas when scanning Lakehouse data?
- How do you operationalise Fabric data quality at scale?
- How does Fabric-native data quality compare to third-party tools?
- UK governance, audit trails, and compliance: what does your team need to address?
- Consultant perspective: the sequence I use on client engagements
- Sources
How does Microsoft Fabric implement and organise data quality?
Fabric’s data quality capability is not a single product. It is a set of interlocking components that each handle a distinct layer of the quality stack.
OneLake and Lakehouse form the storage foundation. All profiling and scanning targets Delta or Iceberg tables stored in OneLake. Parquet files are partially supported but only under specific directory patterns, which matters for teams migrating from Azure Data Lake Storage.
Materialized Lake Views (MLVs) sit above raw Lakehouse tables. They support declarative constraint definitions using CONSTRAINT ... ON MISMATCH DROP or CONSTRAINT ... ON MISMATCH FAIL syntax. According to practitioner guidance on MLV constraints, Fabric surfaces dropped-row counts in lineage and provides an embedded data quality report for monitoring. ON MISMATCH FAIL stops the entire refresh; ON MISMATCH DROP excludes offending rows and continues, counting each row once even when multiple constraints fail simultaneously.
Microsoft Purview contributes three distinct capabilities here: the Data Map (metadata catalogue and lineage), the Unified Catalog (data product management, profiling, and scorecards), and Data Quality scanning (rule authoring, scan execution, and results). These three are connected but administered separately, which is a source of operational friction.
Data Activator handles alerting. You wire it to profiling outputs or scorecard thresholds to trigger notifications when quality metrics breach defined limits.
Monitor hub gives platform owners a consolidated view of pipeline runs, refresh statuses, and scan histories across the workspace.
A useful mental model: profiling and rule-based scans live in Purview (Unified Catalog), declarative checks live in MLVs at the transformation layer, lineage and metadata live in the Data Map, and alerting lives in Data Activator. Scorecards surface in Power BI.
Key terms to align your team on before starting:
- Profiling: statistical analysis of a table’s columns (null rates, distinct counts, value distributions) run by Purview
- Data quality rule: a declarative condition (null check, pattern match, range constraint, referential integrity) applied during a Purview scan
- Data product: a curated, governed unit in the Unified Catalog that groups Lakehouse assets under an owner and SLA
- Purview MSI: the managed service identity Purview uses to authenticate against Fabric for data quality scans (distinct from the service principal used for Data Map scans)
How do you set up Fabric Lakehouse data quality from scratch?
Microsoft Learn documents the full prerequisites and connection requirements for Purview profiling and data quality scans against a Fabric Lakehouse. Work through this sequence before touching rule authoring.
Prerequisites checklist
- Table format: confirm all target tables are Delta or Iceberg. Parquet is only supported under specific directory structures; convert to Delta where feasible.
- Data Map scan completed: Purview must have already scanned the Lakehouse via the Data Map before data quality scans can run. No Data Map scan means no assets to target.
- Data product association: the Lakehouse (or specific tables) must be added to a data product in the Unified Catalog. Data quality scans are scoped to data products, not raw Lakehouse paths.
- Purview Hub enabled: confirm the Purview Hub is activated in your Fabric tenant. This requires a tenant admin to enable it via the Admin API.
- Licensing: Purview Data Quality is a premium capability. Confirm your Microsoft 365 or Purview licence tier covers it before building workflows.
Connection fields to document
Capture these values before configuring any scan. Missing one mid-setup causes failures that are slow to diagnose:
- Tenant ID: your Azure Active Directory tenant GUID
- Workspace ID: the Fabric workspace GUID (visible in the workspace URL)
- Lakehouse ID: the specific Lakehouse GUID within that workspace
- Credential for Data Map scans: a service principal with appropriate Fabric permissions
- Credential for Data Quality scans: the Purview MSI (not the service principal)
- Key Vault name and secret name: if storing credentials in Azure Key Vault (recommended)
- Private Link / VNet flags: whether your Fabric capacity runs behind a managed VNet or private endpoint
- Azure region: confirm Purview and Fabric capacity are in the same region; cross-region scans introduce latency and can fail silently
Permission steps
- In the Fabric workspace, navigate to Workspace settings > Permissions.
- Add the Purview MSI as a Contributor on the workspace. Without Contributor access, data quality scans will authenticate but fail to read table metadata.
- For Data Map scans, grant the service principal Reader access at minimum on the Lakehouse item. Some scan configurations require Contributor.
- If using Azure Key Vault for secrets, grant the Purview MSI Key Vault Secrets User on the relevant vault.
Troubleshooting common setup failures
- Scan returns zero assets: the Data Map scan has not completed or the Lakehouse is not yet associated with a data product. Re-run the Data Map scan and check the data product membership in the Unified Catalog.
- Authentication errors on DQ scan: you are using the service principal instead of the Purview MSI for the data quality step. These are separate credentials for separate scan types.
- VNet blocking scans: if your capacity uses a managed VNet, you need a managed private endpoint from Purview to the Fabric workspace. Check the Private Link configuration in the Azure portal.
- Previously ingested assets inconsistent: assets scanned before the Purview Hub was enabled may show metadata mismatches between the Data Map and Unified Catalog. A full re-scan resolves most cases.
Pro Tip: Create a dedicated test Lakehouse with a small Delta table before configuring scans against production. This lets you validate the full credential chain, MSI permissions, and data product setup without risking production scan quotas or surfacing misleading quality scores on live data.
How do you profile tables, create rules, and run a data quality scan?
With prerequisites in place, the profiling-to-scan workflow follows a consistent sequence.
Step 1: Run a profile. In the Unified Catalog, open the data product containing your Lakehouse table. Select the table and trigger a profile run. Purview analyses column-level statistics: null rates, distinct value counts, min/max ranges, and top-value distributions. This baseline is what rule thresholds are calibrated against.

Step 2: Review profiling outputs. Look for columns with unexpectedly high null rates, distributions that suggest truncation or encoding errors, and date ranges that fall outside expected bounds. These are your highest-priority candidates for rules.
Step 3: Author rules. Purview’s no-code rule editor supports the following types:
- Null constraint: flags or fails rows where a mandatory column is null
- Pattern check: validates string columns against a regex (e.g., UK postcode format
^[A-Z]{1,2}[0-9][0-9A-Z]?s?[0-9][A-Z]{2}$) - Range check: confirms numeric or date values fall within expected bounds
- Referential integrity: validates that values in one column exist in a reference set
For MLV-level checks, the declarative constraint syntax gives you ON MISMATCH DROP for soft enforcement and ON MISMATCH FAIL for hard stops. Use FAIL on critical identity or key columns; use DROP on enrichment columns where partial data is acceptable.
Step 4: Run the scan. Schedule or trigger the data quality scan from the Unified Catalog. Purview evaluates each rule against the table and produces a scorecard.
Step 5: Interpret results. The scorecard shows a pass rate per rule and an aggregate quality score. Prioritise rules tied to downstream reports or ML feature pipelines first.
Pro Tip: Start with three to five rules per table, not thirty. Overly broad rule sets produce alert noise that teams learn to ignore. Use the profiling baseline to set thresholds at the 99th percentile of observed values, then tighten after two weeks of scan data.
What are the known limitations and gotchas when scanning Lakehouse data?
Fabric-native data quality scanning works well within its supported boundaries. Outside those boundaries, teams hit friction that is not always well-documented.
Supported formats:
- Delta tables: fully supported, preferred format for all DQ workflows
- Iceberg tables: supported with some metadata constraints
- Parquet: limited support, only under specific directory naming patterns; not suitable as a primary DQ target
Common gotchas:
- Metadata inconsistency on previously ingested assets: Lakehouse tables scanned before the Purview Hub was enabled can appear differently in the Data Map versus the Unified Catalog. A full re-scan via the Data Map, followed by a schema refresh in the Unified Catalog, resolves most cases.
- VNet and Private Link misconfigurations: managed VNet capacities require a managed private endpoint from Purview. Missing this causes scan jobs to time out with generic authentication errors rather than a clear network message.
- Contributor access misconfiguration: the Purview MSI must hold Contributor on the Fabric workspace, not just Reader. A common mistake is granting Reader, which allows authentication but blocks the metadata reads the DQ scan needs.
- SQL engine statistics latency on large tables: as noted by practitioner analysis of Fabric DQ challenges, the Fabric SQL engine can recalculate statistics beyond Delta logs on very large tables, causing significant query delays during initial scans. Plan maintenance windows for first-run scans on tables above a few hundred million rows.
- No single end-to-end DQ console: Purview DQ, MLV constraints, and pipeline tests each surface results in different places. Coordinating a unified quality view requires deliberate effort across Azure AD, Purview, and Fabric.
Quick remediations:
- Re-run a full Data Map scan, then refresh the schema in the Unified Catalog
- Create a sample Lakehouse with a representative Delta table to isolate credential and format issues
- Convert Parquet sources to Delta before registering them as DQ targets
- For large-table latency, schedule initial scans during off-peak hours and use incremental scan modes once the baseline is established
How do you operationalise Fabric data quality at scale?
Getting a first scan working is straightforward. Keeping quality stable across dozens of data products in production requires deliberate operational design.
Core principles:
- Validation-first pipelines: place MLV constraint checks at the point of ingestion, before data reaches the Gold or serving layer. A row that fails a constraint should never reach a downstream semantic model silently.
- Gate at layer transitions: treat each Medallion layer boundary (Bronze to Silver, Silver to Gold) as a quality gate. A pipeline that moves data forward without a passing quality check is a liability.
- Observable contracts for data products: define explicit quality SLAs per data product (e.g., “null rate on CustomerID < 0.1%, freshness lag < 4 hours”) and surface them in Power BI scorecards.
KPIs and metrics to track:
- Profile drift: change in null rate, distinct count, or value distribution between scan runs
- Violation rate: percentage of rows failing each rule per scan
- Dropped rows: count surfaced in MLV lineage for
ON MISMATCH DROPconstraints - Freshness lag: time between the last successful data load and the current scan timestamp
Wire Data Activator alerts to violation rate thresholds and freshness lag breaches. For scorecard visibility, publish Purview quality scores to a Power BI report that data stewards review daily.
Google’s SRE guidance on data integrity recommends a defence-in-depth approach: tiered validators where daily lightweight checks catch the most dangerous invariants, and deeper checks run weekly or monthly. A tiered validation strategy also reduces alert noise, which is one of the fastest ways to erode team trust in a quality programme.
Operational checklist:
- Assign a named data owner per data product, responsible for remediation SLAs
- Define run cadence: daily incremental scans for high-value operational tables, weekly full scans for historical or reference data
- Document remediation runbooks: for each rule type, specify the expected fix, the team responsible, and the SLA (e.g., critical violations resolved within 4 hours, warnings within 48 hours)
- Review scan history monthly and retire rules that consistently pass without ever firing, replacing them with checks on newly identified risk areas
An incident at a production database, documented by Glitchive, illustrates what happens when out-of-band validation and runbooks are absent: a single undetected write operation caused irreversible data loss during a code freeze. Out-of-band validators and documented runbooks are not optional at enterprise scale.
How does Fabric-native data quality compare to third-party tools?
The decision between Fabric-native checks and dedicated DQ tooling comes down to where your constraints live, how your pipelines are built, and what your governance obligations require.
| Dimension | Fabric-native (Purview + MLVs) | Engineering-native test frameworks (Great Expectations, dbt tests) | Enterprise DQ platforms (Informatica IDMC, Ataccama) |
|---|---|---|---|
| Best for | Teams already in the Microsoft ecosystem needing profiling and governance | Data engineers running pipeline-native tests in CI/CD | Organisations with MDM, complex stewardship workflows, or multi-source governance |
| Fabric/Purview integration | Native; no additional connectors | Requires custom pipeline steps or Fabric notebooks | Connector-based; varies by vendor and version |
| Setup complexity | Moderate; MSI/SPN configuration, Purview Hub activation | Low for dbt; moderate for Great Expectations at scale | High; enterprise licencing, metadata sync, and admin overhead |
| Validation style | Post-hoc scans (Purview) + declarative constraints (MLVs) | Pipeline-native tests run at transformation time | Metadata-driven rules with UI-based stewardship and approval workflows |
| Governance and MDM | Lineage, data products, scorecards; no MDM | Minimal; test results in CI/CD logs | Full MDM, master data stewardship, policy enforcement |
| Scalability | Good for Fabric-scale workloads; large-table latency is a known constraint | Scales with pipeline infrastructure; no native Fabric scorecard | Enterprise-grade; designed for multi-domain, multi-source environments |
When to use Fabric-only: your data estate lives primarily in OneLake, your tables are Delta-format, and your governance requirements are met by Purview lineage and scorecards. This is the right starting point for most teams new to structured data quality management.
When to pair Fabric with dbt tests or Great Expectations: your transformation pipelines already run in dbt or you want test results integrated into a CI/CD gate. These frameworks add pipeline-native validation that Purview scans do not replicate. They do not replace Purview profiling; they complement it.
When to adopt an enterprise DQ platform: your organisation requires MDM, cross-domain data stewardship, or complex rule libraries that span multiple source systems. Informatica IDMC and Ataccama both offer Fabric connectors, but the integration overhead is significant. Soda (formerly Soda SQL / Soda Core) sits between these tiers: it supports Fabric via Spark and offers a no-code rule interface closer to Purview’s, making it a practical middle option for teams that need more than MLVs but less than full MDM.
The practitioner analysis of Fabric DQ challenges confirms that Purview Data Quality’s no-code rule creation and AI-generated rule suggestions are genuinely useful, but operationalising them at enterprise scale typically requires pairing with pipeline-native tests and a metadata-driven runbook for remediation.
UK governance, audit trails, and compliance: what does your team need to address?
UK data teams operating in Microsoft Fabric face two overlapping obligations: the requirements of ISO/IEC 5259-3:2024 for auditable data quality management systems, and UK GDPR obligations when profiling or processing personal data.
ISO/IEC 5259-3 and what to implement in Fabric
ISO/IEC 5259-3:2024 specifies data quality management requirements for analytics and ML, with emphasis on auditable lifecycle evidence. To meet those principles in Fabric:
- Maintain scan history and rule change logs in the Unified Catalog as audit artefacts
- Record MLV constraint definitions and refresh logs as version-controlled artefacts (store in a Git-connected Fabric workspace)
- Use Data Map lineage to document data provenance from source to serving layer
- Define and document critical data elements (CDEs) per data product, aligned with the DAMA-NL DQMS framework, which enumerates policy, quality objectives, supplier management, and monitoring as core DQMS elements
UK GDPR checklist for Fabric profiling
When profiling tables that contain personal data, the following apply:
- Lawful basis: confirm a lawful basis under UK GDPR Article 6 before profiling personal data columns. Profiling is processing; it requires the same legal basis as any other processing activity.
- Data minimisation: profile only the columns necessary for the quality objective. Avoid running full-column profiling on tables containing special category data unless strictly required.
- Provenance and access controls: document who can trigger scans and view profiling outputs. Use Fabric workspace roles and Purview access policies to restrict profile result visibility to authorised data stewards.
- Retention of audit artefacts: retain scan logs, rule definitions, and lineage records for a period consistent with your data retention policy and any applicable regulatory requirement.
- Data subject rights: if profiling surfaces personal data in quality reports (e.g., sample values in profiling outputs), confirm those outputs are excluded from data subject access request scope or anonymised before storage.
Provenance-based integrity tracking, as explored in research on provenance-based integrity proofs, supports auditability without requiring third-party auditors, which aligns well with Fabric’s native lineage capabilities in the Data Map.
Consultant perspective: the sequence I use on client engagements
The most common mistake I see teams make is trying to implement data quality everywhere at once. They spend weeks configuring Purview scans across twenty Lakehouse tables, author fifty rules, and then discover that half the rules fire constantly because the thresholds were set without profiling data to calibrate them. The quality programme collapses under its own alert volume before it delivers a single remediation.
The sequence that works in practice is much narrower to start.

First, run a quick discovery: identify the two or three data products that cause the most downstream pain when they contain errors. These are your starting targets, not your entire estate. Second, profile those tables and spend time reading the profiling output before authoring a single rule. The profile tells you what “normal” looks like; rules without that baseline are guesses. Third, add three to five MLV constraints on the highest-risk columns, using ON MISMATCH DROP initially so you can observe dropped-row counts without blocking pipeline runs. Fourth, associate the Lakehouse with a data product in the Unified Catalog, run a Purview DQ scan, and review the scorecard with the data owner, not just the engineering team.
The pitfalls I encounter most often: missing the Contributor grant for the Purview MSI (causes cryptic authentication failures), ignoring Delta format requirements (Parquet sources silently fail to scan), and setting ON MISMATCH FAIL on every constraint before understanding the data’s natural variance. That last one stops pipeline refreshes entirely and creates an emergency that undermines confidence in the whole programme.
When to bring in external support: if your organisation needs MDM, cross-domain governance, or a Fabric implementation that spans multiple workspaces and business units, the coordination overhead across Azure AD, Purview, and Fabric grows quickly. That is where Datataal’s Fabric consulting services add the most value: scoping the right starting point, configuring the credential chain correctly the first time, and building the operational runbooks that keep quality stable after go-live.

If your team is starting a Fabric data quality implementation or hitting friction with Purview scan configuration, I work directly with data platform teams to get these workflows production-ready. Hire a Microsoft Fabric consultant and we can scope your engagement in a single call.
Sources
The sources below are organised by type: official documentation, standards, and practitioner commentary.
Official documentation:
- Data Quality for Fabric Lakehouse in Unified Catalog
- ISO/IEC 5259-3:2024 – Artificial intelligence — Data quality for analytics and machine learning (ML) — Part 3: Data quality management requirements and guidelines
- Data Quality as Code in Fabric: Declarative Checks on Materialized Lake Views – EduDataSci
- Data Quality Management System – Framework
Standards:
Practitioner commentary: