mssql-to-pyspark-migration

Inventory the procedure's parameters, temp tables, cursors, transactions, and DML statements, then map each construct to Spark SQL or DataFrame API equivalents. The Skill generates a reusable Python function with dry_run support, table_map for environment-specific names, and a validation plan.

T-SQL MERGE maps directly to Delta Lake MERGE INTO with WHEN MATCHED and WHEN NOT MATCHED clauses. Deduplicate the source first using ROW_NUMBER or QUALIFY, since Delta MERGE fails when multiple source rows match one target row.

Cursors should be rewritten as set-based operations using joins, aggregations, window functions, or MERGE rather than row-by-row Python loops. Iterative loops are acceptable only for small datasets or unavoidable side effects, and the reason must be documented.

Databricks does not provide multi-statement transaction scope like SQL Server BEGIN TRAN/COMMIT/ROLLBACK. Delta Lake gives ACID guarantees per operation, so multi-step logic needs idempotent task design, compensation logic, or Delta time travel for rollback.

Interpolating user input into spark.sql strings creates SQL injection vulnerabilities, the top risk in MSSQL-to-Databricks conversions. Use DataFrame API filters with col() comparisons, typed temp views for parameterized MERGE, and whitelist validation for identifiers instead.

GRANT SELECT maps directly, while INSERT, UPDATE, and DELETE map to GRANT MODIFY on Unity Catalog tables. GRANT EXECUTE has no direct equivalent and is replaced by workspace, notebook, and job access controls; DENY is replaced by least-privilege grants, row filters, and column masks.