Michael Tall — Header
Engineer inspecting industrial streaming sensor

Ship Subsecond Fabric Real Time Streaming With OneLake for Engineers

Microsoft Fabric Real-Time Intelligence is now generally available, giving teams a single workload for ingesting, processing, storing, analysing and acting on streaming data. It brings together Real-Time Hub, Eventstreams, Eventhouse, Real-Time Dashboards and Activator into one governed environment. For production workloads, that means subsecond query performance, built-in CI/CD support and native connectors, without stitching together separate PaaS services.


TL;DR:

  • Real-Time Intelligence is ideal for workloads requiring seconds or minutes delay, such as equipment telemetry, fraud detection, and live operational dashboards.
  • Eventstreams support Kafka-compatible endpoints, enabling integration with existing producers without rewriting data sources.
  • Eventhouse excels with low-latency queries, anomaly detection, and organizing time-series data, while OneLake handles long-term historical storage.
  • Activator rules require manual updates if their respective query logic changes to prevent mismatched alerts and actions.
  • Capacity planning should include monitoring, retention management, and autoscaling to avoid system slowdowns and unexpected costs.

Datataal
datataal.com
Build Reliable Real-Time Data Foundations
Datataal designs scalable Microsoft Fabric architectures that connect streaming data, governance, analytics, and reporting capabilities.

Visit Datataal

Table of Contents

What Real-Time Intelligence is and when to use it

Real-Time Intelligence sits inside Microsoft Fabric as the workload responsible for anything that cannot wait for a nightly batch job. It stores data natively in OneLake, so streaming data and your lake-based historical data live in the same governed estate rather than two separate systems with separate security models.

The practical question most teams face is not “can Fabric do real-time,” but “does this workload need it.” Batch and scheduled pipelines remain the right choice for reporting that tolerates hours of latency, such as monthly finance close or weekly sales roll-ups. Real-Time Intelligence earns its complexity when decisions depend on data that is seconds or minutes old.

Typical triggers for adopting the real-time path include:

  • Equipment or sensor telemetry where a delayed alert means a missed failure window.
  • Fraud or anomaly detection where minutes of lag defeats the purpose.
  • Operational dashboards that executives or shop-floor staff watch live rather than review the next morning.
  • Event-driven automations, such as triggering a workflow the moment a threshold is crossed.

The workload reached general availability with Real-Time Hub, Eventstream, Eventhouse, Real-Time Dashboards and Activator all included, along with enterprise features such as APIs and CI/CD support. GA status matters here: it signals Microsoft’s commitment to supported upgrade paths and SLAs, which is the detail most procurement and architecture review boards actually ask about before signing off a production dependency. It does not mean every feature inside the workload is equally mature, and we will flag the previews as we go.

Core components: Hub, Eventstreams, Eventhouse, Activator and dashboards

Five components do the work, and each has a distinct job in the pipeline.

Real-Time Hub is the discovery and catalogue layer. It lists every stream, source and destination in your tenant, so engineers can find an existing Eventstream rather than rebuilding a connector that already exists elsewhere in the organisation.

Eventstreams handle ingestion and transformation. They offer no-code source connectors, a code-first SQL operator for teams that want explicit control, and a growing set of destinations including Eventhouse, Activator and custom endpoints. Eventstreams also support Kafka-compatible endpoints, which matters if you already have producers writing to a Kafka protocol and do not want to rewrite them.

Eventhouse is the storage and query engine purpose-built for time-series and event data. It organises records by arrival time, supports KQL natively, and runs anomaly detection in place on streaming data without needing to move it elsewhere first.

Activator converts conditions into actions. It watches streams or KQL querysets and fires alerts or downstream automations when a rule is met, rather than requiring someone to be staring at a dashboard.

Real-Time Dashboards render the data with low latency, built on KQL queries against Eventhouse, suited to operational monitoring rather than quarterly reporting.

Together, a typical flow looks like: a source lands in an Eventstream, gets reshaped with a no-code transform or SQL operator, writes into Eventhouse, and then feeds both a dashboard and an Activator rule simultaneously.

Five-component Fabric streaming flow

Pro Tip: Build your Eventstream transforms as derived streams rather than one monolithic pipeline, so you can reuse a cleaned, filtered stream across multiple downstream destinations without duplicating logic.

Stream processing: SQL operator, no-code transforms and Spark notebooks

Three processing paths cover most transformation needs, and the right one depends on who maintains the logic and how complex it gets.

  1. No-code operators inside an Eventstream handle filtering, field mapping, aggregation and simple joins visually, and output a derived stream that other pipelines can reuse without rebuilding the transform.
  2. The SQL operator gives you a single-statement, code-first transformation model. It supports windowing, TIMESTAMP BY clauses, and writing to multiple destinations from one query, but it does not support chaining multiple SQL operators in one processing path, so complex multi-stage logic needs to be split across separate Eventstreams or moved to Spark.
  3. Spark notebook integration suits transformations too complex for either of the above. Fabric auto-generates PySpark snippets and provides a secure, Fabric-optimised Kafka adapter to read Eventstreams directly into a notebook with token-based authentication, which removes the usual friction of wiring up Spark Structured Streaming by hand. Prefer this path when you need custom machine learning scoring, complex stateful aggregations, or logic that is easier to express in PySpark than in KQL or SQL.

Our guide to Eventstream setup walks through configuring each of these in a working pipeline.

Pro Tip: Start with the no-code operators and the SQL operator before reaching for Spark. Most production streams need filtering, windowing and routing, not custom code, and the simpler path is cheaper to maintain.

Storage, querying and OneLake interaction

Eventhouse and OneLake solve different problems, and the architecture decision is mostly about which one holds the data you are querying most often.

Eventhouse is the right home for time-series and event data that needs low-latency KQL queries, native anomaly detection and arrival-time ordering. It is built for the “what just happened” question, not for long-term archival reporting.

OneLake holds your historical, governed lake data, and the two are designed to work together rather than in competition. OneLake Query Acceleration lets you combine historical OneLake data with live Eventhouse streams in a single query, with Microsoft citing improvements of up to 50 times for certain query patterns that would otherwise require moving data between systems first. In practice, this is the pattern to reach for whenever an alert or dashboard needs streaming context plus historical baseline, such as comparing current machine vibration against last quarter’s average.

Retention and caching policy is the other lever worth understanding early. Eventhouse supports a caching tier for data you query frequently and a standard storage tier for data kept mainly for compliance or rare lookups. Setting retention too aggressively on the cache tier is a common cause of unexpectedly slow dashboards once a stream has been running for a few months, because queries fall back to the slower standard tier without anyone noticing until latency complaints start.

Real-time analytics and visualisation: KQL, dashboards and Copilot

KQL is the query language doing most of the work once data lands in Eventhouse. It is built for time-series patterns: filtering by time window, computing rolling aggregates, and detecting deviations from a baseline are all native operations rather than workarounds. Eventhouse’s anomaly detection runs in place on streaming data, so a KQL query can surface outliers without a separate scoring step.

Real-Time Dashboards are built directly on these KQL queries, and are designed for operational monitoring rather than executive reporting. Typical components include:

  • Time-series charts tracking a metric over a rolling window.
  • Tile-based KPI summaries refreshed on a short interval.
  • Drill-through links from a summary tile into the underlying KQL queryset.

For teams more comfortable in Power BI than KQL, our Power BI consulting services extend these same Eventhouse sources into familiar report formats.

Copilot adds a natural-language layer on top of this, letting analysts ask questions about a stream without writing KQL from scratch, and surfacing anomaly suggestions during exploration. It is a starting point for drafting a query, not a replacement for understanding what the underlying KQL is actually doing, particularly once a dashboard needs to be trusted for operational decisions.

Alerts, automation and Activator-driven actions

Activator is the component that turns a condition into something that happens automatically, rather than something a person has to notice. Rules can be threshold-based, pattern-based across a time window, or driven directly by a KQL queryset result.

Once triggered, a rule can send an alert, kick off a Fabric pipeline, or hand off to an external workflow such as Power Automate, which is where real-time monitoring turns into real-time operations rather than just real-time reporting.

One operational caution is easy to miss: Activator snapshots a KQL queryset at the point the rule is created, so editing the original queryset later does not automatically update the running rule. If you change the underlying query logic, you need to update or recreate the Activator rule itself, otherwise it keeps evaluating the old definition while the dashboard built on the new one shows something different. This mismatch is a common source of confusing alert behaviour in teams that treat Activator rules as “set and forget.”

Operations, capacity planning and cost drivers

Eventhouse capacity behaves differently from a typical batch warehouse, and planning for it upfront avoids most of the surprises teams hit in their first few months of production use.

  • Capacity units and autoscale flex with load, but Eventhouse UpTime and a capacity scheduler let you guarantee a baseline level of capacity rather than relying purely on reactive scaling.
  • Throttling runs across three levels: proactive, reactive and extreme reactive, each progressively more aggressive about limiting throughput to protect the system. Scheduling a minimum baseline capacity reduces how often you hit the reactive tiers during predictable peak periods.
  • Storage tiers split into a cache tier for frequently queried recent data and a standard tier for older, infrequently accessed data, with cost rising as more data stays in the faster cache tier.
  • Retention policy should match query patterns, not just compliance minimums: keeping six months of raw telemetry in the cache tier when dashboards only ever query the last 30 days is a common source of avoidable cost.

Enabling workspace monitoring is worth doing early rather than retrofitting later, since it generates a dedicated monitoring Eventhouse that gives you queryable metrics across your deployments instead of piecing together capacity health from separate views.

Reference architectures and real-world use cases

A handful of patterns cover most production deployments. The connected factory pattern ingests sensor telemetry through Event Hubs or MQTT, enriches it with contextual data such as machine specifications or maintenance history pulled from OneLake, and runs analytics in Eventhouse to flag deviations in near real time. Predictive maintenance architectures that mix Eventhouse time-series storage with OneLake historical context are the main route to achieving subsecond response times for industrial telemetry rather than running everything against a single store.

For teams working with physical sensor networks and facility monitoring, our partner BeyondSensor covers how sensor-driven monitoring feeds into operational dashboards, a pattern that maps closely onto the Eventhouse and Activator combination described above.

Fraud detection and agent telemetry follow a similar shape: ingest, enrich, detect, act, just with different latency and volume profiles. A payment fraud stream typically needs sub-second scoring and a tight Activator rule, while agent or application telemetry tolerates slightly looser latency but much higher event volume.

Derived streams earn their place whenever more than one downstream system needs the same cleaned data, and DeltaFlow is worth reaching for specifically when your source is CDC events that need reshaping before they are analytics-ready.

Implementation checklist and best practices

Moving from a working prototype to a production streaming pipeline comes down to a short list of disciplines that are easy to skip under deadline pressure and expensive to retrofit later.

  1. Lock down authentication early. Use EntraID for every connector and grant the least privilege each service principal actually needs, rather than reusing a broad admin credential across sources.
  2. Set up Git integration before building pipelines, not after. Fabric’s deployment pipelines and Git integration support real-time artifacts, but migrating an already-built Eventstream into source control later is more work than starting there.
  3. Respect the same-workspace limitation. Some real-time artifacts and their dependencies need to live in the same workspace to deploy cleanly, so plan your workspace structure before splitting projects across teams.
  4. Enable workspace monitoring so you have queryable metrics on capacity and throttling rather than discovering a problem from a user complaint.
  5. Rebuild Activator rules after editing their source queryset, since edits to the underlying KQL do not propagate automatically.

Our phased implementation guide for IT teams walks through sequencing these steps across a realistic project timeline.

Pro Tip: Treat your first Activator rule and its KQL queryset as a matched pair. If one changes, check the other before calling the change complete.

Practitioner notes and evidence from Michael Taal

As a Microsoft Fabric consultant, I design streaming architectures that lean on the medallion pattern inside Fabric: raw events land in a bronze layer, get cleaned and conformed into silver, and surface as materialised gold views for dashboards and Activator rules. This keeps streaming pipelines auditable and lets teams rebuild a layer without touching the others.

Three-layer streaming data architecture

Typical deliverables include Eventstream pipelines wired to Eventhouse, Real-Time Dashboards built on top, and governance that keeps access scoped per workspace. Our blog covers more detail on specific pipeline designs and outcomes from past engagements.

Lessons from deploying Fabric real-time

The mistakes I see most often are predictable: teams underestimate Eventhouse capacity needs under real load, skip workspace monitoring until something breaks, and forget to rebind Activator rules after editing a queryset.

My advice is to start with one mission-critical stream rather than migrating everything at once. Prove the pattern, measure latency and cost against expectations, then expand. A scoped engagement usually gets a team from prototype to a trustworthy production stream faster than an open-ended internal project, mainly because the capacity and governance decisions get made deliberately rather than inherited by accident.

— Michael

How Datataal can help with Fabric real-time streaming

We design and build Microsoft Fabric streaming architectures end to end, from Eventstream and Eventhouse setup through to the dashboards and Activator rules that turn data into action. Clients work directly with a Microsoft Fabric specialist for the whole engagement, with no junior staff or management layers between the architecture decisions and the people making them.

Datataal

Services on offer include:

  • Microsoft Fabric architecture and implementation, covering Eventstream, Eventhouse and capacity planning.
  • Power BI and real-time dashboard development on top of your streaming data.
  • Data engineering and governance, including Azure Synapse to Fabric migration.
  • Training and ongoing support so your team can maintain what we build.

A typical engagement starts with a short scoping conversation about your streaming sources and latency requirements, then moves into a fixed-scope pilot before wider rollout. Pricing for Microsoft Fabric consulting services starts from $20 per hour. Book a consultation to talk through your use case.

FAQ

What is Fabric vs Databricks?

Microsoft Fabric is a unified analytics platform covering ingestion, real-time streaming, data engineering, warehousing and Power BI reporting within one governed environment built on OneLake. Databricks is a data engineering and machine learning platform centred on Spark, and the two are often compared on data engineering workloads rather than on real-time streaming specifically, where Fabric’s Real-Time Intelligence workload is purpose-built.

What is an example of real-time data?

Sensor telemetry from factory equipment, financial transaction streams used for fraud scoring, and live application or website event logs are common examples of real-time data. Each is characterised by needing analysis within seconds or minutes of the event occurring, rather than tolerating a batch delay.

Is Fabric IQ generally available?

This article covers Real-Time Intelligence in Microsoft Fabric, which reached general availability with Real-Time Hub, Eventstream, Eventhouse, Real-Time Dashboards and Activator included. We do not have publicly listed details on a separately branded “Fabric IQ” product, so that naming is not addressed here.

What is real-time streaming data?

Real-time streaming data is a continuous flow of events, such as sensor readings or transaction records, processed as it arrives rather than collected and analysed later in a batch. In Microsoft Fabric, this flow typically moves through an Eventstream into Eventhouse for storage and query, then on to dashboards or Activator for alerting.

Sources

Microsoft Fabric blog