skills.rest
jerrylin96
jerrylin96COMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-autocleaning

Profiles and cleans BigQuery and GCS data using Dataplex scans and SQL transformations.

9|Updated Jul 8, 2026
Search Tags:#data quality#data cleaning#bigquery#gcs#data profiling#sql transformation#dataplex
One-click install
npx skills add https://github.com/jerrylin96/dotgemini --skill data-autocleaning-jerrylin96
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-autocleaning
Source: https://github.com/jerrylin96/dotgemini/tree/main/skills/data-autocleaning
Command: npx skills add https://github.com/jerrylin96/dotgemini --skill data-autocleaning-jerrylin96

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Moving or ingesting data from BigQuery or Google Cloud Storage often introduces hidden quality issues like malformed dates, garbage values, inconsistent units, and schema mismatches. This Skill automates profiling, cleansing, and verification so pipelines built on Dataform, dbt, or BigQuery start from trustworthy data. ## Core Features & Use Cases - Dataplex-Based Profiling: Runs automated data profile scans (with dynamic sampling for tables over 1M rows) before and after transformation to drive data-driven cleaning decisions. - SQL Transformation Generation: Produces cleansing SQL covering schema alignment, type conversion with SAFE.PARSE functions, JSON/ARRAY/STRUCT handling, and unit normalization. - Quality Review Protocol: Validates transformations by profiling a temporary sample output table and comparing column-level metrics against the source profile, iterating until anomalies are resolved. - Use Case: When asked to copy a GCS dataset into a BigQuery destination table, the Skill profiles the source, generates cleaning SQL matched to the destination schema, and verifies the result with a post-transformation Dataplex scan. ## Quick Start Ask the agent to clean and load a BigQuery or GCS source into a destination table, for example: profile and clean the data in project.dataset.raw_orders and load it into project.dataset.clean_orders.

Frequently Asked Questions about data-autocleaning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean data when moving it from GCS to BigQuery?▼

Create an external table over the GCS source, run a Dataplex data profile scan to detect anomalies, then generate cleansing SQL transformations aligned to the destination schema. Verify the result by profiling a temporary sample output table and comparing it to the source profile.

How to profile BigQuery tables with Dataplex before transformation?▼

Run the dataplex_scanner.py script with table IDs in project.dataset.table format, a specific region like us-central1, and an output directory. It handles concurrent scans, dynamic sampling for tables over 1M rows, and saves results as JSON files.

Does Dataplex scanning support multi-region locations like us?▼

No, Dataplex data profile scans require a specific Google Cloud region such as us-central1. Multi-region locations like us are not supported and will cause the scan to fail.

Can I skip Dataplex profiling and use ad-hoc BigQuery queries instead?▼

No, the workflow mandates Dataplex profiling for both pre-transformation and post-transformation quality review. Ad-hoc BigQuery profiling is only used as a fallback when the user explicitly denies scan execution approval.

Why does my BigQuery array transformation fail with NULL elements?▼

BigQuery arrays cannot contain NULL values, so SAFE_CAST failures inside arrays cause errors. Filter out NULLs after casting using ARRAY_FILTER(array_column, e -> e IS NOT NULL) before aggregating elements back into an array.

What happens if the user denies approval for the Dataplex scan?▼

The denial is documented in the implementation plan and walkthrough evidence sections instead of job IDs. Verification then proceeds manually using bq sample queries to confirm transformations resolved the detected anomalies.

Related Skills

Explore More Data & Analytics Skills→
danielpratsdanielprats

uploadCarto

Upload GeoParquet files from a local directory to Google Cloud Storage and import them into CARTO Data Warehouse.

Community
Intermediate
daemon-blockint-techdaemon-blockint-tech
7

analytics-data-engineer

Guide dbt-style analytics project structuring, governance, and CI.

Community
Advanced
databricks-solutionsdatabricks-solutions
5

anomaly-detection

Detect schema-level anomalies for table freshness and completeness in Databricks Unity Catalog.

Official
Intermediate
apacheapache
595

datafusion-python

Write DataFusion DataFrame and SQL queries with Apache Arrow integration in Python.

Official
Advanced