processing-s3-uploads-with-step-functions

Deploy an event-driven Step Functions workflow routing S3 uploads to Lambda or Fargate by file size.

Updated Jul 1, 2026
Ready to run
npx skills add https://github.com/sakicodes/BuildFestHackathon26 --skill processing-s3-uploads-with-step-functions-sakicodes
💬 Prompt for ChatGPT / Claude Chat▼
Please help me install this Agent Skill.
Skill: processing-s3-uploads-with-step-functions
Source: https://github.com/sakicodes/BuildFestHackathon26/tree/main/.agents/skills/processing-s3-uploads-with-step-functions
Command: npx skills add https://github.com/sakicodes/BuildFestHackathon26 --skill processing-s3-uploads-with-step-functions-sakicodes

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires boto3, and includes scripts (resource) and references (resource) components.

What problem does it solve? Processing S3 uploads often requires different compute depending on file size: small files fit Lambda's limits while large files need containerized Fargate tasks. Manually wiring EventBridge, Step Functions, Lambda, ECS, ECR, VPC networking, and scoped IAM roles is error-prone and time-consuming. ## Core Features & Use Cases - Size-Based Routing: A Step Functions Choice state sends files over 6 MB to a Fargate task and smaller files to a Lambda function. - Complete Infrastructure Provisioning: Creates the S3 bucket with EventBridge notifications, ECR repository, ECS cluster and task definition, VPC security group, CloudWatch log group, and scoped IAM roles via AWS CLI. - Built-In Monitoring and Security: Configures a Dead Letter Queue for failed EventBridge invocations, CloudWatch alarms on execution failures, KMS encryption, and scoped security group egress. - Use Case: A data pipeline receives user uploads ranging from kilobytes to gigabytes; this workflow automatically processes small files in Lambda and large files in Fargate without manual intervention. ## Quick Start Ask the AI to deploy the S3 upload processing workflow by providing a bucket name, AWS region, ECR repository name, and Step Functions state machine name.

Frequently Asked Questions about processing-s3-uploads-with-step-functions

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I route S3 uploads to Lambda or Fargate based on file size?▼

Enable EventBridge notifications on the S3 bucket, trigger a Step Functions state machine on Object Created events, and use a Choice state on $.detail.object.size. Files over 6 MB run an ECS Fargate task; smaller files invoke Lambda.

How do I trigger Step Functions from S3 upload events?▼

Enable EventBridge notifications on the bucket with put-bucket-notification-configuration, then create an EventBridge rule matching source aws.s3 and detail-type Object Created. Add the state machine as a target with an IAM role allowing states:StartExecution.

What prerequisites are needed to deploy this Step Functions workflow?▼

You need AWS CLI v2 configured with valid credentials, Python 3.12 for the Lambda runtime, and Docker to build and push the Fargate container image to ECR. A default VPC with two subnets is also required.

Why is my EventBridge rule not triggering the state machine?▼

Verify EventBridge notifications are enabled on the bucket with get-bucket-notification-configuration, confirm the rule exists with aws events describe-rule, and check the target has the correct state machine ARN and IAM role.

When should I not use Step Functions for S3 file processing?▼

Skip this approach when all files fit Lambda's limits, since a direct S3 to Lambda trigger is simpler. It is also unsuitable for real-time streaming workloads, where Kinesis is the appropriate service.

Why is my Fargate task stuck in PROVISIONING state?▼

Fargate tasks stall when subnets lack auto-assign public IP or the route table has no internet gateway route for 0.0.0.0/0. Verify the security group allows outbound HTTPS and DNS traffic for image pulls and logging.