skills.rest
Aradotso
AradotsoOFFICIAL
Explore all skills by this author →
Website𝕏@aradotsoGitHub

harvard-artifacts-data-engineering-pipeline

Extract artifact data from the Harvard Art Museums API into SQL databases.

5|1|Updated May 16, 2026
Search Tags:#SQL#ETL#data analysis#Streamlit#data engineering
One-click install
npx skills add https://github.com/Aradotso/data-skills --skill harvard-artifacts-data-engineering-pipeline
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: harvard-artifacts-data-engineering-pipeline
Source: https://github.com/Aradotso/data-skills/tree/main/skills/harvard-artifacts-data-engineering-pipeline
Command: npx skills add https://github.com/Aradotso/data-skills --skill harvard-artifacts-data-engineering-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires streamlit, pandas, requests, mysql-connector-python, plotly, python-dotenv, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a comprehensive end-to-end solution for building ETL pipelines and analytics dashboards using the Harvard Art Museums API, empowering users to collect, transform, store, and analyze artifact data efficiently.

Core Features & Use Cases

  • ETL Pipeline: Automates data extraction, transformation, and loading from the Harvard Art Museums API.
  • Database Design: Implements normalized tables for structured data storage.
  • SQL Analytics: Executes predefined analytical queries for in-depth data analysis.
  • Visualization: Generates interactive dashboards with Streamlit and Plotly.
  • Use Case: Ideal for data engineers and analysts looking to create detailed analytics of museum collections and artifacts.

Quick Start

Run the ETL pipeline with the provided script to fetch, transform, and load data from the Harvard Art Museums API into a SQL database.

Frequently Asked Questions about harvard-artifacts-data-engineering-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an ETL pipeline to extract and load API data into a SQL database?▼

To build an ETL pipeline with API data, you extract records using HTTP requests, transform the payload with pandas, and load it into normalized SQL database tables. This Skill automates that exact workflow using the Harvard Art Museums API and Python scripts.

Can I use Streamlit and Plotly to create an interactive dashboard for SQL database analytics?▼

Yes, you can generate interactive dashboards for SQL database analytics by querying the database with connectors and visualizing results using Streamlit and Plotly. This Skill provides predefined analytical queries to feed data directly into the dashboard interface.

What is the best way to structure normalized SQL tables for museum artifact data?▼

Structuring normalized SQL tables for museum artifact data involves designing schemas that separate entities like objects, classifications, and artists into related tables. This Skill implements normalized database tables specifically for structured storage of Harvard Art Museums API records.

Do I need a MySQL database to run this ETL pipeline and analytics dashboard?▼

Yes, a MySQL database is required because the ETL pipeline utilizes the mysql-connector-python dependency to load and query the transformed artifact data. You must configure your database credentials in a python-dotenv environment file before execution.

How does Python handle API data extraction and transformation for large museum collections?▼

Python handles API data extraction and transformation for large museum collections by using the requests library to fetch paginated records and pandas to clean and structure the JSON payloads. The transformed data is then ready for structured SQL database loading.

Are there limitations when using Streamlit for data engineering visualization workflows?▼

Streamlit visualizations in data engineering workflows are limited to local or web-based interactive dashboards rather than automated batch reporting. This Skill focuses on interactive dashboard generation rather than scheduled static report distribution for artifact analytics.

Related Skills

Explore More Data & Analytics Skills→
AradotsoAradotso
5

harvard-artifacts-collection-analytics-etl

Extract, transform, and load Harvard Art Museums API data into a SQL database.

Official
Advanced
AradotsoAradotso
5

harvard-artifacts-etl-pipeline

Automate ETL pipelines for Harvard Art Museums data into SQL databases.

Official
Advanced
apacheapache
595

datafusion-python

Write DataFusion DataFrame and SQL queries with Apache Arrow integration in Python.

Official
Advanced
AradotsoAradotso
5

harvard-art-museum-data-engineering

Automate ETL from the Harvard Art Museums API into SQL databases with Streamlit dashboards.

Official
Advanced