skills.rest
HTTPArchive
HTTPArchiveOFFICIAL
Explore all skills by this author →
Website𝕏@HTTPArchiveGitHub

optimize-storage-costs

Identify and remove dead-end and unused BigQuery tables using Masthead Data lineage analysis.

7|4|Updated Aug 26, 2024
Search Tags:#data lineage#cost optimization#data governance#bigquery#storage management#table cleanup
One-click install
npx skills add https://github.com/HTTPArchive/dataform --skill optimize-storage-costs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: optimize-storage-costs
Source: https://github.com/HTTPArchive/dataform/tree/main/.agents/skills/optimize-storage-costs
Command: npx skills add https://github.com/HTTPArchive/dataform --skill optimize-storage-costs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps reduce cloud spending by identifying and removing BigQuery tables that are no longer used or are redundant, thereby optimizing storage costs.

Core Features & Use Cases

  • Identify Waste: Pinpoints "dead-end" tables (updated but not read) and "unused" tables (no activity).
  • Cost Reduction: Provides actionable steps to safely remove identified tables, leading to direct cost savings.
  • Use Case: A data team notices a spike in their BigQuery bill. They use this Skill to find and eliminate several old, unreferenced tables that were consuming significant storage.

Quick Start

Use the optimize-storage-costs skill to query for tables with over $10 in potential monthly savings.

Frequently Asked Questions about optimize-storage-costs

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find unused BigQuery tables to reduce storage costs?▼

To reduce BigQuery storage costs, this Skill identifies dead-end tables that are updated but never read, and unused tables with no activity, using Masthead Data lineage analysis to pinpoint waste.

What is a dead-end table in BigQuery data governance?▼

A dead-end table in BigQuery data governance is a table that is actively updated but never read downstream. Identifying these tables via lineage analysis helps eliminate redundant storage expenses.

Do I need Masthead Data to identify abandoned tables in BigQuery?▼

Yes, you need Masthead Data agent v0.2.7 or higher and BigQuery access. The agent provides the data lineage analysis required to query insights and safely drop abandoned tables.

How do I clean up BigQuery storage by dropping tables safely?▼

To clean up BigQuery storage, the Skill queries for tables with over $10 in potential monthly savings and provides actionable steps to safely drop the identified unreferenced and dead-end tables.

What is the best way to optimize BigQuery storage hygiene?▼

The best way to optimize BigQuery storage hygiene is analyzing data lineage to find tables with no activity. This approach targets unreferenced tables directly, ensuring direct cost savings.

Related Skills

Explore More Data & Analytics Skills→
ssmleossmleo

datalineage-bigquery-asset-impact-analysis

Analyze downstream impact of BigQuery asset changes using Data Lineage API.

Community
Advanced
gke-labsgke-labs
42

gke-cost-analysis

Analyze GKE infrastructure costs from BigQuery billing export datasets.

Official
Intermediate
truongnattruongnat

data-engineering-pro

Orchestrate ETL/ELT pipelines and data warehouse architectures with Kafka, Spark, and Flink.

Community
Advanced
hreis00hreis00

gcp-bigquery

Optimize BigQuery queries, schemas, partitioning, clustering, and cost controls.

Community
Advanced