data-processing-workflows

What problem does it solve?

This Skill clarifies the complex data processing pipelines within Polibase, ensuring data is collected, processed, and matched correctly. It automates the understanding of intricate dependencies and execution orders for tasks like minutes processing, web scraping, and speaker matching, preventing data inconsistencies and operational errors.

Core Features & Use Cases

• Minutes Processing Flow: Guides through the sequential steps of dividing PDF minutes, extracting speakers, and matching them with politicians.
• Web Scraping & GCS Integration: Explains how to scrape data from websites, upload to Google Cloud Storage, and integrate with the processing pipeline.
• Conference Member Extraction: Details a multi-stage process for extracting, matching, and creating affiliations for conference members, including LLM-based fuzzy matching and manual review.
• Use Case: When you need to ingest new meeting minutes, this skill provides the exact sequence of commands and checks to ensure the PDF is processed, speakers are extracted, and matched to politicians, ultimately populating the database with accurate, linked data.

Quick Start

I need to process a new PDF meeting minute. Guide me through the entire data processing workflow, from minutes division to speaker matching.