spark-authoring-cli

What problem does it solve?

Develop Microsoft Fabric Spark/data engineering workflows and write code in Fabric Notebook cells with intelligent routing to specialized resources. Provides workspace/lakehouse management, notebook code authoring (PySpark, Scala, SparkR, SQL), and routes to: data engineering patterns, development workflow, or infrastructure orchestration. Use when the user wants to: (1) manage Fabric workspaces and resources, (2) write or debug code in notebook cells, (3) use notebookutils, (4) develop notebooks and PySpark applications, (5) design data pipelines, (6) provision infrastructure as code. Triggers: "develop notebook", "data engineering", "workspace setup", "pipeline design", "infrastructure provisioning", "Delta Lake patterns", "Spark development", "lakehouse configuration", "write notebook code", "notebookutils", "notebook cell", "PySpark notebook", "%%sql cell", "%%configure", "fabric notebook", "run notebook", "notebook deployment".

Core Features & Use Cases

• Comprehensive CLI for Spark authoring, workspace/lakehouse management, and notebook code authoring
• Routes to data engineering patterns, development workflow, and infrastructure orchestration
• Supports PySpark, Scala, SparkR, and SQL with correct lakehouse access and notebookutils
• Use cases include creating workspaces, provisioning lakehouses, authoring notebooks, and deploying Spark jobs

Quick Start

Use the spark-authoring-cli to scaffold a Spark notebook in a Fabric workspace and start writing PySpark/Scala/SparkR/SQL code with guided routing.