skills.rest
bbvch-ai
bbvch-aiOFFICIAL
Explore all skills by this author →
GitHub

rclone-guide

Configure rclone cloud storage connectors for Dagster data pipelines.

9|2|Updated Nov 29, 2024
Search Tags:#oauth#data pipeline#cloud storage#dagster#rclone#file synchronization#connector setup
One-click install
npx skills add https://github.com/bbvch-ai/aihub-core --skill rclone-guide
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rclone-guide
Source: https://github.com/bbvch-ai/aihub-core/tree/main/.claude/skills/rclone-guide
Command: npx skills add https://github.com/bbvch-ai/aihub-core --skill rclone-guide

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the manual guesswork and configuration overhead of connecting cloud storage backends to data pipelines using rclone, removing the need to manually debug RC API settings, environment variables, or OAuth authentication for 70+ supported cloud services.

Core Features & Use Cases

  • Pre-built connector helpers: One-click configuration for common backends including OneDrive, Google Drive, S3, SharePoint, and Azure Blob Storage via pre-validated environment variable loaders.
  • Full integration reference: Complete guidance for using RcloneClient, RcloneResource, and RcloneIOManager to integrate rclone with Dagster pipeline orchestration.
  • Troubleshooting support: Clear, actionable fixes for common rclone errors including connection failures, authentication mismatches, expired OAuth tokens, and filter pattern issues.
  • Use Case: If you need to connect your team's Google Drive to the company data lake pipeline, this Skill walks you through setting up the required OAuth credentials, environment variables, and verifying the connection without manual configuration.

Quick Start

Use the rclone-guide skill to set up a new OneDrive cloud storage connector for your data pipeline by providing your OAuth app client ID, client secret, and desired remote name.

Frequently Asked Questions about rclone-guide

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure rclone cloud storage connectors for a data pipeline?▼

To configure rclone cloud storage connectors, use environment-variable based remote configuration to programmatically define credentials and settings for 70+ backends like S3 and Google Drive without manual setup.

Why does my rclone OAuth authentication fail when connecting to Google Drive?▼

Rclone OAuth authentication fails when expired tokens or mismatched credentials occur, requiring actionable fixes to connection settings and environment variables to restore cloud storage access.

Can I integrate rclone with Dagster for pipeline orchestration?▼

Yes, you can integrate rclone with Dagster by configuring RcloneClient, RcloneResource, and RcloneIOManager to establish observable assets and handle file metadata listing for hash-based change detection.

Does rclone support SharePoint and Azure Blob Storage backends for file synchronization?▼

Yes, rclone supports SharePoint and Azure Blob Storage among 70+ cloud storage backends, enabling file synchronization and remote connector setup through pre-validated environment variable loaders.

What is the best way to programmatically set up rclone remotes for dynamic use cases?▼

The best way to set up rclone remotes dynamically is through programmatic configuration and environment variables, which enables automated testing and rapid backend deployment without manual CLI intervention.

How do I fix rclone filter pattern issues during file synchronization?▼

Fix rclone filter pattern issues by adjusting connection settings and verifying filter rules, resolving common configuration errors that disrupt file synchronization across cloud storage backends.

Related Skills

Explore More Software Engineering Skills→
pulzzepulzze

interactor-auth

Authenticates backend systems to Interactor platform using OAuth2 client credentials.

Official
Intermediate
bbvch-aibbvch-ai
9

dagster-pipelines

Implement Dagster pipeline architectures with asset factories, ops, and resources.

Official
Advanced
noahwins-ngnoahwins-ng
1

dagster-expert

Guide Dagster pipeline development, asset management, and dg CLI operations.

Community
Advanced
fivetranfivetran
70

build-connector

Automate end-to-end Fivetran Connector SDK development for unsupported APIs.

Official
Advanced