data-autocleaning

Dataplex profiling scans BigQuery and GCS source tables to detect null rates, value distributions, and hidden anomalies before generating SQL-based cleansing logic. This approach profiles data first, then applies safe conversions and schema mapping to resolve quality issues and produce schema-aligned outputs.

Yes, Dataplex profiling runs first to uncover distinct counts, null rates, and value distributions in BigQuery and GCS data. These profiling results guide the safe SQL-based cleansing transformations, ensuring schema alignment and resolving detected quality issues before finalizing any pipeline changes.

The best way to validate cleaning logic is rebuilding a temporary output table, profiling the transformed result with Dataplex, and comparing it against the original source. This post-transformation quality review confirms the cleansing SQL fixed detected issues without introducing new data regressions.

Yes, the cleaning process handles ingestion, copying, moving, and extraction workflows by applying schema mapping and struct alignment rules. It uses safe SQL conversions, JSON parsing, and array handling to ensure BigQuery and GCS data meets the required schema structure during pipeline validation.

Generate cleansing SQL by running Dataplex scans to profile the source data for sparse fields and hidden anomalies. The Skill then creates safe SQL transformations to parse JSON, handle arrays, and align structs, producing a cleaned, schema-aligned output table ready for downstream use.