spark-authoring-cli

POST to /v1/workspaces/{workspaceId}/items with type Lakehouse using az rest and the resource https://api.fabric.microsoft.com. Set creationPayload.enableSchemas to true to organize tables into bronze, silver, and gold schemas.

Use getDefinition to retrieve the notebook, decode the base64 payload, modify the ipynb JSON cells, re-encode, and upload with updateDefinition. Each line in a cell source array must end with a newline character to prevent code merging.

Spark SQL MLVs support incremental refresh when source tables have Change Data Feed enabled, while PySpark MLVs always use full refresh. Use PySpark only when you need UDFs or complex Python logic that SQL cannot express.

The request body must be flat JSON with name, driverMemory, driverCores, executorMemory, and executorCores. Wrapping the body in a payload object or sending only a kind field causes the HTTP 500 error.

Window functions, SELECT DISTINCT, RIGHT or FULL OUTER JOIN, ORDER BY, LIMIT, and non-deterministic functions like current_date force full refresh. Rewrite with INNER JOIN, GROUP BY, and deterministic filters, moving ranking logic downstream.

Query job instances from the last five minutes before submitting a new run, and monitor an existing job instead of creating a duplicate. After submission, capture the job instance ID immediately and poll status rather than retrying the POST.