LLM Integration Patterns

Implements multi-provider LLM routing, streaming, tool calling, and failover for AI agent SaaS applications.

Updated Apr 2, 2026

Search Tags:#rate limiting #llm #tool calling #streaming #failover #ai sdk #litellm

Ready to run

npx skills add https://github.com/khiwniti/carbonscope --skill llm-integration-patterns-khiwniti

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: LLM Integration Patterns
Source: https://github.com/khiwniti/carbonscope/tree/main/ai-agent-saas-expert/skills/llm-integration-patterns
Command: npx skills add https://github.com/khiwniti/carbonscope --skill llm-integration-patterns-khiwniti

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires litellm, ai, @ai-sdk/react, zod, @upstash/redis, @upstash/ratelimit, @workflow/ai, and includes references (resource) and examples (resource) components.

What problem does it solve? Building production LLM features requires handling provider outages, streaming responses, tool calling, cost tracking, and rate limiting across multiple vendors, which is complex to implement correctly from scratch. ## Core Features & Use Cases - Multi-Provider Routing: Configure LiteLLM or Vercel AI Gateway with automatic failover between Anthropic Claude and OpenAI GPT models, including retry logic and router configuration. - Streaming & Tool Calling: Implement server-side streaming with the Vercel AI SDK, define typed tools with Zod schemas, and render tool calls in React using AI Elements. - Production Hardening: Add Redis caching, token-based cost tracking, per-user rate limiting with Upstash, exponential backoff retries, and durable agents via Workflow DevKit. - Use Case: You are building a chat feature in a Next.js SaaS app and need Claude as the primary model with GPT fallback, streaming UI, and per-user daily request limits. ## Quick Start Set up a streaming chat endpoint in my Next.js app using the AI SDK with Claude as the primary model and GPT as a fallback.

Frequently Asked Questions about LLM Integration Patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up multi-provider LLM failover in Python?▼

Use LiteLLM's completion or acompletion functions with a fallbacks parameter listing alternative models like GPT after Claude. The LiteLLM Router class also supports named model groups, retry counts, and timeout configuration for self-hosted deployments.

How to stream LLM responses in a Next.js app?▼

Use the Vercel AI SDK's streamText function in a route handler and return result.toUIMessageStreamResponse(). On the client, the useChat hook from @ai-sdk/react consumes the stream and updates messages in real time.

LiteLLM vs Vercel AI Gateway for LLM routing?▼

LiteLLM suits self-hosted VM backends needing full routing control, 100+ provider support, and no vendor lock-in. AI Gateway fits Vercel-hosted frontends wanting automatic failover and unified billing without managing API keys.

How do I add tool calling to an AI SDK agent?▼

Define tools with the tool function using Zod schemas for inputSchema and outputSchema, plus an execute function. Pass them to streamText or an Agent with a stopWhen condition like stepCountIs to limit tool call iterations.

How do I rate limit LLM requests per user?▼

Use Upstash Ratelimit with a sliding window limiter backed by Redis, keyed by user ID. Check the limit before processing each request and return HTTP 429 when the daily quota is exceeded.

When should I use durable agents for LLM workflows?▼

Use DurableAgent from Workflow DevKit when agent executions must survive crashes and redeployments, such as long-running research tasks. It provides observable steps, automatic retry on failure, and pause/resume support.