Michael Tall — Header
Engineer authoring a Fabric Spark notebook

Engineers: Production Ready Spark Notebooks in Fabric, CI/CD & Cost

A Spark notebook in Microsoft Fabric is an interactive, managed Apache Spark authoring surface. It lets you run distributed transforms, prototype machine learning models and connect directly to OneLake without provisioning a cluster. I use notebooks for ad hoc transforms, ML experiments and diagnostic runs, and reach for pipelines or Spark Job Definitions once logic needs to run repeatedly in production.

TL;DR:

  • Fabric notebooks are ideal for quick exploration, prototyping machine learning, and ad hoc transforms without provisioning clusters, but can become slow if they accumulate large outputs or many cells.
  • They support multiple languages—including PySpark, Scala, SQL, and R—using magic commands, and configuration changes must be placed in the first cell with a session restart for them to take effect.
  • Autoscale billing and reserved capacity options affect costs based on workload variability, with reserved capacity offering around a 40% discount for steady jobs.
  • Performance can be improved using native execution engine acceleration, early filtering, and staged transforms, while profiling tools help monitor resource use and identify bottlenecks.
  • For production workflows, convert stable notebook logic into Spark Job Definitions or pipelines, and follow disciplined promotion, lineage tracking, and cleanup practices for CI/CD governance.
Datataal
Build a More Scalable Fabric Architecture
Datataal helps organizations design and implement governed data architectures that simplify analytics, reporting, and real-time business insight.

Explore Datataal’s services

Table of Contents

Develop, author and manage Fabric notebooks

A Fabric notebook is built from cells, each holding code or markdown, that you run independently or in sequence. This cell model is what makes notebooks useful for exploration: you can test a transform on a small dataframe, inspect the output, then adjust the next cell without rerunning everything above it. Fabric notebooks are the primary code item for developing Apache Spark jobs and experiments in the platform, and they support import and export in formats including .ipynb and source .py files.

Collaboration works through workspace roles and permissions, with version history and AutoSave keeping a record of changes so you can roll back an experiment that went wrong. The Monaco editor underneath the notebook interface brings IntelliSense, code completions and a variable explorer, which speeds up development by surfacing dataframe schemas and variable states without extra print statements.

A few habits keep notebooks manageable as they grow:

  • Keep each notebook scoped to one purpose: exploration, a specific transform or a diagnostic check, not all three.
  • Move small utility functions into the resource folder rather than repeating them across notebooks.
  • Avoid saving large dataframe outputs inline; write results to a table or file instead of leaving them in cell output.
  • Export stable logic to .py files once it moves from prototype to repeatable use.

These rules matter more than they look. A notebook that accumulates months of exploratory cells and full dataframe printouts becomes slow to open, slow to save and hard for a second engineer to follow.

Languages, magic commands and session configuration

Fabric notebooks support PySpark, Scala, SQL and SparkR within the same notebook, switched cell by cell using magic commands. Microsoft Learn documents these magics as the mechanism for mixing languages without splitting logic across separate items.

  • %%pyspark runs the cell in Python against the Spark session.
  • %%spark runs Scala.
  • %%sql runs Spark SQL directly against a lakehouse table.
  • %%sparkr runs R.
  • %run executes another notebook inline, useful for sharing setup code or utility functions across a project.

%%configure sets Spark session properties such as vCores, driver and executor memory, the pool to use and custom Spark properties. A typical block sets the number of executors and memory per executor before any data loads, so the session starts with the right shape rather than resizing mid-run.

Pro Tip: Place %%configure in the first cell of the notebook and restart the session after changing it. A configuration set midway through a run is ignored until the next session starts.

Performance, profiling and cost: capacity, starter pools, NEE and Autoscale Billing

Spark workloads in Fabric draw on workspace capacity by default, with starter and custom pools billed only while a session is active. Microsoft Learn’s billing and utilisation guidance explains how this capacity consumption is attributed and reported to Azure Cost Management, which matters when several teams share the same capacity.

Autoscale Billing for Spark, now generally available, offers a pay-as-you-go model that isolates Spark jobs from shared Fabric capacity, which suits bursty or exploratory workloads well. Stable, predictable production jobs are usually better served on reserved capacity, where cost stays consistent rather than fluctuating with usage.

Reserved capacity versus Autoscale Billing comparison

Starter pools give you a fast way to test logic before committing to a large custom pool. A medium starter pool, sized around 8 vCores and 64 GB of memory, is enough to surface parallelism issues or data skew before you scale up.

For profiling, three tools cover most needs:

  • Spark History Server for reviewing completed job stages and identifying slow tasks.
  • Resource Usage UI for live memory and CPU consumption during a run.
  • Capacity Metrics app for tracking utilisation across the whole workspace over time.

The Native Execution Engine (NEE) accelerates Spark execution for supported operations by running them through a native vectorised engine rather than the standard JVM path, which reduces runtime on eligible workloads without changing your code. Combined with right-sized executor counts and reduced shuffles, it’s one of the more effective levers for cutting both runtime and cost on repeated jobs.

Reserved capacity offers roughly a 40% discount for steady, predictable workloads compared with pay-as-you-go pricing, which makes the reserved versus Autoscale decision largely about how variable your Spark usage actually is. Our guide to Microsoft Fabric cost goes into the reservation trade-offs in more detail.

In practice, the biggest wins come from reducing shuffles by filtering early, staging transforms instead of chaining everything into one wide job, and setting autoscale limits that match your actual peak load rather than a guess.

Orchestration and automation: scheduling notebooks and pipelines

A notebook that works well interactively still needs a reliable way to run on a schedule or as part of a larger job. Fabric gives you a few options depending on how much control you need.

  1. Use the Job Scheduler API or the Items REST API to trigger parameterised notebook runs and monitor their status programmatically.
  2. Use %run to call another notebook inline when you need shared setup or utility logic without duplicating it.
  3. Use notebookutils.runMultiple() to run several small notebooks in parallel rather than sequentially, which cuts total wall-clock time for independent tasks.
  4. Return exit codes or values from a notebook run so a downstream pipeline step can branch conditionally on success, failure or a specific outcome.
  5. Design each job to be idempotent, so a retry after a failure doesn’t duplicate rows or corrupt a partially written table.

A master and worker pattern, where one orchestrating notebook calls several smaller worker notebooks, tends to scale better than one monolithic script, and it makes retries far easier to reason about, as outlined in the scalable AI automation architecture for UK enterprises. Our guide to Fabric data pipelines covers how to wire these patterns into a full pipeline.

Limits, common failures and troubleshooting

Fabric notebooks carry a few hard limits worth knowing before you hit them mid-project. Microsoft’s documentation lists a 32 MB cap on notebook content, a limit of 256 code cells per notebook and dataframe output size and row caps that can cause UI lag well before those hard ceilings.

  • A notebook approaching the 32 MB content cap will slow down on save and open; split logic into smaller notebooks or move heavy output elsewhere.
  • Session timeouts default to 20 minutes of inactivity, and paused capacity can extend a cold start; keep sessions active with scheduled runs if idle time is a problem.
  • Overly complex single queries often fail or stall; refactor them into staged transforms across intermediate tables instead of one wide operation.
  • Copilot diagnostics and the Fix with Copilot flow in the notebook UI can summarise a failed cell’s error and suggest a fix, which shortens root-cause time on common Spark errors.

Pro Tip: When a notebook starts lagging in the editor before you’ve even run a cell, check cell count and total content size first. It’s usually a size limit, not a Spark problem.

Developer productivity: tooling, IDEs and reproducible workflows

Authoring notebook logic in VS Code, with a proper Git workflow, gives you version control and code review that the notebook UI alone doesn’t provide. Exporting stable logic to modular .py files, rather than leaving everything as notebook cells, makes that code testable and reusable outside the notebook.

  • Keep shared utility code in the resource folder, referenced by multiple notebooks rather than copied into each one.
  • Rely on version history and AutoSave for quick rollbacks, but resolve collaboration conflicts by agreeing on one owner per notebook during active development.
  • Write small unit tests for utility functions and integration smoke tests for staged pipeline steps, rather than testing only inside the notebook.
  • Treat any notebook destined for production as a candidate for conversion into a Spark Job Definition once its logic stabilises.

Practitioner checklist and CI/CD patterns from a Fabric consultant

Notebooks earn their place in a production workflow only when the path from development to deployment is deliberate. I promote notebook code through dev, test and prod workspaces using the Items API, treating each promotion as a controlled step rather than a manual copy.

  • Track lineage carefully so a rerun doesn’t create duplicate nodes in downstream tables.
  • Convert stable notebook logic into a Spark Job Definition or pipeline step once it runs the same way every time.
  • Run a short cleanup pass after each session: clear temporary storage, confirm output tables match expected row counts, and capture profiling metrics before closing the session.
  • Keep dev, test and prod workspace promotion documented, even briefly, so the next engineer can follow it.

Pro Tip: Capture a baseline run’s profiling metrics before optimising anything. Without a baseline, you’re guessing whether a change actually helped.

Our Fabric implementation guide covers this promotion pattern in more depth for teams building out a full CI/CD process.

When Fabric notebooks are the right tool

Fabric notebooks are strongest where integration speed matters more than deep engineering control: fast exploration, lightweight ML prototyping and transforms that feed straight into Power BI through OneLake. For heavy, sustained Spark engineering, dedicated platforms are often the better fit, and hybrid architectures combining both are common rather than exceptional. My own rule is simple: one purpose per item. Notebooks for exploration, pipelines and Spark Job Definitions for anything that needs to run the same way, unattended, every day.

— Michael

How Datataal can help with Fabric notebooks and architecture

Getting from a working notebook to a production-ready Spark workload takes more than good code. It takes the right capacity choice, a CI/CD path that doesn’t rely on manual copying, and tables structured so Direct Lake serves them cleanly to Power BI. I work directly with organisations on exactly this, from initial Microsoft Fabric consulting services through to ongoing support, without junior staff or extra management layers between you and the work.

Datataal

That covers architecture design, data engineering, and migrating existing Azure Synapse workloads into Fabric where notebook and pipeline logic needs rebuilding rather than lifting as-is. If your team is comfortable with the fundamentals but wants a second set of eyes on capacity sizing or a CI/CD pattern that will hold up in production, that’s usually the point to bring in outside help rather than building it from scratch. Book a discovery call through the Microsoft Fabric consulting services page to talk through where your notebooks and pipelines currently stand.

Sources

Most notebook work starts with a read and ends with a write, and Fabric gives you several paths for both. Spark’s native reader handles CSV, Parquet and Delta tables directly from a lakehouse, and a right-click on any file in the lakehouse explorer inserts a load cell with the path already filled in. Pandas works the same way for smaller, in-memory analysis where a full Spark session is unnecessary overhead.

OneLake underpins all of this by storing table data once in Delta format, and Direct Lake reads that same data directly into a Power BI semantic model without a separate extract or refresh step. That matters for teams building dashboards on top of Spark-transformed tables: the curated table a notebook writes becomes the same table Direct Lake serves to Power BI, with no duplication in between. Our guide to Fabric data connectors covers the ingestion side of this in more depth.

A few patterns worth following:

This exploratory-to-curated flow keeps notebooks readable and keeps Direct Lake serving tables that are genuinely production-ready rather than half-finished experiments.

FAQ

Are there Databricks in fabric?

No, Databricks is a separate product and not part of Microsoft Fabric. Fabric has its own native Spark notebooks and Spark Job Definitions, and some organisations run both platforms side by side depending on workload needs.

What are fabric notebooks?

Fabric notebooks are a web-based interactive item for authoring Apache Spark jobs and experiments, built from code and markdown cells. They support PySpark, Scala, SQL and SparkR within the same notebook via magic commands.

What is a spark notebook?

A Spark notebook is an interactive document that combines code cells, markdown and output in one file, designed for exploring data and building distributed transforms with Apache Spark. In Fabric specifically, it connects directly to lakehouse tables and OneLake without separate cluster setup.

What are the limitations of Fabric notebook?

Fabric notebooks carry hard limits including a 32 MB cap on notebook content, a maximum of 256 code cells, and caps on dataframe output size and row count. Sessions also time out by default after 20 minutes of inactivity, which can interrupt long-running interactive work.

Microsoft Fabric blog