skills.rest
ruskibeats
ruskibeatsCOMMUNITY
Explore all skills by this author →
GitHub

stream-jsonl-to-postgres

Stream gzipped JSONL files into Postgres using asyncpg COPY.

Updated May 15, 2026
Search Tags:#postgresql#batch processing#data import#jsonl#asyncpg
One-click install
npx skills add https://github.com/ruskibeats/t1d --skill stream-jsonl-to-postgres
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: stream-jsonl-to-postgres
Source: https://github.com/ruskibeats/t1d/tree/main/.pi/skills-archive/stream-jsonl-to-postgres
Command: npx skills add https://github.com/ruskibeats/t1d --skill stream-jsonl-to-postgres

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires asyncpg, and includes scripts (resource) components.

What problem does it solve?

This Skill addresses the challenge of importing large gzipped JSONL files into Postgres databases efficiently, avoiding the need for full disk decompression or loading everything into memory.

Core Features & Use Cases

  • Streamlined Import: Directly streams gzipped JSONL files into Postgres using asyncpg COPY.
  • Batch Processing: Utilizes async batch processing to handle large data volumes without overloading memory.
  • Use Case: Ideal for large-scale data imports like Open Food Facts exports, log files, or OpenStreetMap data without the overhead of pandas or intermediate file formats.

Quick Start

To import a gzipped JSONL file into a table named 'my_table', use the command: import_jsonl_to_postgres 'path_to_file.jsonl' 'my_table'

Frequently Asked Questions about stream-jsonl-to-postgres

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I import a large gzipped JSONL file into Postgres without running out of memory?▼

Batch processing with asyncpg COPY streams gzipped JSONL files directly into Postgres without full decompression or loading everything into memory.

What's the best way to batch process JSONL data into a PostgreSQL database?▼

Using asyncpg's COPY command for asynchronous batch processing streams JSONL data into PostgreSQL efficiently without intermediate file storage or heavy memory usage.

Do I need pandas to import OpenStreetMap or Open Food Facts JSONL exports into Postgres?▼

No, you do not need pandas; asyncpg COPY streams large-scale JSONL exports directly into Postgres, avoiding the overhead of pandas or intermediate file formats.

How does asyncpg COPY handle streaming compressed JSONL files into a database?▼

asyncpg COPY streams compressed JSONL files by asynchronously batching data directly into Postgres without requiring intermediate file storage or full decompression.

Can I use this method for large-scale log file imports without intermediate storage?▼

Yes, asyncpg COPY streams gzipped JSONL log files directly into Postgres using asynchronous batch processing without intermediate file storage.

Related Skills

Explore More Data & Analytics Skills→
Aqsagull99Aqsagull99

sqlmodel-skill

Create reusable SQLModel-based ORM layers with async sessions for Python applications.

Community
Intermediate
dfujiwaradfujiwara

psql

Query a local Kindle notes PostgreSQL database with SQL or natural language.

Community
Advanced
LeandroBenjaminLLeandroBenjaminL

db-admin

Manage database administration tasks for PostgreSQL, MySQL, Redis, and SQLite.

Community
Advanced
youyinnnyouyinnn
1

polars

Perform in-memory data manipulation with lazy evaluation and parallel execution.

Community
Advanced