skills.rest
Flintmere
FlintmereOFFICIAL
Explore all skills by this author →
GitHub

bulk-catalog-sync

Stream Shopify catalog JSONL results into Postgres with chunked writes.

Updated Apr 18, 2026
Search Tags:#streaming#etl#shopify#jsonl#postgres#bulk-sync#bulk operation
One-click install
npx skills add https://github.com/Flintmere/flintmere --skill bulk-catalog-sync
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bulk-catalog-sync
Source: https://github.com/Flintmere/flintmere/tree/main/.claude/skills/bulk-catalog-sync
Command: npx skills add https://github.com/Flintmere/flintmere --skill bulk-catalog-sync

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Ingest and stream a Shopify merchant's full catalog without loading it into memory.

Core Features & Use Cases

  • Streaming bulk results via JSONL to avoid OOM issues.
  • Chunked writes (500 items per transaction) with upsert behavior to Postgres.
  • Use cases include initial migrations, handling very large catalogs, and refactoring the streaming parser.

Quick Start

Start the bulk-catalog-sync process to stream your entire catalog in chunks and persist scoring records to Postgres.

Frequently Asked Questions about bulk-catalog-sync

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I import a large Shopify catalog without running out of memory?▼

Streaming Shopify bulk operation results via JSONL avoids out-of-memory issues by parsing records incrementally and writing them in chunked transactions to Postgres.

How do I stream Shopify bulk operation JSONL results to Postgres?▼

Stream Shopify bulk operation JSONL results to Postgres by continuously parsing the stream and upserting records in chunked transactions of 500 items per write.

Does Shopify bulk catalog sync handle expired URLs and malformed JSONL lines?▼

Yes, robust error handling manages expired URLs, malformed JSONL lines, and concurrent bulk operations to ensure reliable data streaming and ingestion.

Can I use Shopify bulk operations for initial catalog migrations to Postgres?▼

Yes, streaming Shopify bulk operations via JSONL is designed for initial catalog migrations, writing data to Postgres in chunked transactions with upsert behavior.

What is the best way to handle timeouts when syncing a very large Shopify catalog?▼

Using the bulkOperationRunQuery API to stream JSONL results in chunks prevents memory overload and timeouts when syncing very large Shopify catalogs.

Related Skills

Explore More Software Engineering Skills→
PostergullyPostergully

setup

Automate GBrain provisioning, AGENTS.md injection, and initial data import.

Community
Advanced
hasnaxyzhasnaxyz

react-native

Render JSON UI specs into native React Native mobile interfaces.

Official
Advanced
bmsull560bmsull560

agentic-ux

Design agent-driven UI patterns for streaming responses, tool logs, and confirmations.

Community
Advanced
40RTY-ai40RTY-ai
174

shopify-admin-duplicate-sku-barcode-detector

Detect duplicate SKU and barcode identifiers across Shopify product variants via GraphQL.

Official
Intermediate